AUTHENTICATED , 
U.S. GOVERNMENT ^ 
INFORMATION ' 


THE SCIENCE OF SECURITY, 
PARTS I AND II 


HEARINGS 

BEFORE THE 

SUBCOMMITTEE ON INVESTIGATIONS AND 
OVERSIGHT 

COMMITTEE ON SCIENCE AND 
TECHNOLOGY 

HOUSE OF REPRESENTATIVES 

ONE HUNDRED ELEVENTH CONGRESS 

FIRST SESSION 

JUNE 25, 2009 
AND 

NOVEMBER 17, 2009 


Serial No. 111-38 
and 

Serial No. 111-63 

Printed for the use of the Committee on Science and Technology 



Available via the World Wide Web: http://www.science.house.gov 


U.S. GOVERNMENT PRINTING OFFICE 
50-324PDF WASHINGTON : 20010 


For sale by the Superintendent of Documents, U.S. Government Printing Office 
Internet: bookstore.gpo.gov Phone: toll free (866) 512—1800; DC area (202) 512-1800 
Fax: (202) 512-2104 Mail: Stop IDCC, Washington, DC 20402-0001 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00001 Fmt5011 Stmt 5011 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 


COMMITTEE ON SCIENCE AND TECHNOLOGY 


HON. BART GORDON, Tennessee, Chair 

RALPH M. HALL, Texas 
F. JAMES SENSENBRENNER JR., 


JERRY F. COSTELLO, Illinois 

EDDIE BERNICE JOHNSON, Texas 

LYNN C. WOOLSEY, California 

DAVID WU, Oregon 

BRIAN BAIRD, Washington 

BRAD MILLER, North Carolina 

DANIEL LIPINSKI, Illinois 

GABRIELLE GIFFORDS, Arizona 

DONNA F. EDWARDS, Maryland 

MARCIA L. FUDGE, Ohio 

BEN R. LUJAN, New Mexico 

PAUL D. TONKO, New York 

PARKER GRIFFITH, Alabama 

STEVEN R. ROTHMAN, New Jersey 

JIM MATHESON, Utah 

LINCOLN DAVIS, Tennessee 

BEN CHANDLER, Kentucky 

RUSS CARNAHAN, Missouri 

BARON P. HILL, Indiana 

HARRY E. MITCHELL, Arizona 

CHARLES A. WILSON, Ohio 

KATHLEEN DAHLKEMPER, Pennsylvania 

ALAN GRAYSON, Florida 

SUZANNE M. KOSMAS, Florida 

GARY C. PETERS, Michigan 

VACANCY 


Wisconsin 

LAMAR S. SMITH, Texas 
DANA ROHRABACHER, California 
ROSCOE G. BARTLETT, Maryland 
VERNON J. EHLERS, Michigan 
FRANK D. LUCAS, Oklahoma 
JUDY BIGGERT, Illinois 
W. TODD AKIN, Missouri 
RANDY NEUGEBAUER, Texas 
BOB INGLIS, South Carolina 
MICHAEL T. MCCAUL, Texas 
MARIO DIAZ-BALART, Florida 
BRIAN P. BILBRAY, California 
ADRIAN SMITH, Nebraska 
PAUL C. BROUN, Georgia 
PETE OLSON, Texas 


Subcommittee on Investigations and Oversight 


HON. BRAD MILLER, North Carolina, Chair 


STEVEN R. ROTHMAN, New Jersey 
LINCOLN DAVIS, Tennessee 
CHARLES A. WILSON, Ohio 
KATHY DAHLKEMPER, Pennsylvania 
ALAN GRAYSON, Florida 
BART GORDON, Tennessee 


PAUL C. BROUN, Georgia 
BRIAN P. BILBRAY, California 
VACANCY 


RALPH M. HALL, Texas 


DAN PEARSON Subcommittee Staff Director 
EDITH HOLLEMAN Subcommittee Counsel 
JAMES PAUL Democratic Professional Staff Member 
DOUGLAS S. PASTERNAK Democratic Professional Staff Member 
KEN JACOBSON Democratic Professional Staff Member 
ALEX MATTHEWS Research Assistant 
TOM HAMMOND Republican Professional Staff Member 
JANE WISE Research Assistant 


(II) 


VerDate Oct 09 2002 


09:19 Mar 17, 2010 Jkt 000000 PO 00000 


Frm 00002 


Fmt 5904 Sfmt 5904 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



CONTENTS 

The Science of Security, Part I: Lessons Learned in Devel- 
oping, Testing and Operating Advanced Radiation Mon- 
itors 

June 25, 2009 

Page 

Witness List 2 

Hearing Charter 3 

Opening Statements 

Statement by Representative Brad Miller, Chairman, Subcommittee on Inves- 
tigations and Oversight, Committee on Science and Technology, U.S. House 

of Representatives 8 

Written Statement 9 

Statement by Representative Paul C. Broun, Ranking Minority Member, Sub- 
committee on Investigations and Oversight, Committee on Science and 

Technology, U.S. House of Representatives 10 

Written Statement 11 

Panel I: 

Mr. Gene Aloise, Director, Natural Resources and Environment, U.S. Govern- 
ment Accountability Office (GAO) 

Oral Statement 12 

Written Statement 14 

Biography 22 

Dr. Micah D. Lowenthal, Director, Nuclear Security and Nuclear Facility 
Safety Program, Nuclear and Radiation Studies Board, National Research 
Council, The National Academies 

Oral Statement 22 

Written Statement 24 

Biography 28 

Discussion 

Prioritizing Security Needs 29 

Decision-making: Processes and Timelines 30 

ASP in Context: The Global Nuclear Detection Architecture 31 

Management of ASP Testing and Deployment 32 

Justifying the Cost of the Advanced Spectroscopic Portal Program 33 

Panel II: 

Dr. William K. Hagan, Acting Deputy Director, Domestic Nuclear Detection 
Office, Department of Homeland Security 

Oral Statement 35 

Written Statement 37 

Biography 40 

Mr. Todd C. Owen, Acting Deputy Assistant Commissioner, Office of Field 
Operations, U.S. Customs and Border Protection, Department of Homeland 
Security 

Oral Statement 41 

Written Statement 42 

(III) 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00003 Fmt5904 Sfmt5904 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



IV 


Page 


Mr. Todd C. Owen, Acting Deputy Assistant Commissioner, Office of Field 
Operations, U.S. Customs and Border Protection, Department of Homeland 
Security — Continued 

Biography 46 

Discussion 47 

Appendix: Additional Material for the Record 

Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radi- 
ation Detection Portal Monitors, but Preliminary Results Show Limits of 
the New Technology, Report to Congressional Requesters, U.S. Government 

Accountability Office, GAO-09-655, May 2009 54 

Prepublication Copy, Evaluating Testing, Costs, and Benefits of Advanced 
Spectroscopic Portals for Screening Cargo at Ports of Entry, Interim Report 
(Abbreviated Version), Committee on Advanced Spectroscopic Portals, Nu- 
clear and Radiation Studies Board, Division of Earth and Life Studies, 
National Research Council of the National Academies 86 

The Science of Security, Part II: Technical Problems 
Continue to Hinder Advanced Radiation Monitors 

November 17, 2009 

Witness List 178 

Hearing Charter 179 

Opening Statements 

Statement by Representative Brad Miller, Chairman, Subcommittee on Inves- 
tigations and Oversight, Committee on Science and Technology, U.S. House 

of Representatives 184 

Written Statement 185 

Statement by Representative Paul C. Broun, Ranking Minority Member, Sub- 
committee on Investigations and Oversight, Committee on Science and 

Technology, U.S. House of Representatives 186 

Written Statement 187 

Witnesses: 

Mr. Gene Aloise, Director, Natural Resources and Environment, U.S. Govern- 
ment Accountability Office (GAO) 

Oral Statement 193 

Written Statement 195 

Biography 199 

Dr. Timothy M. Persons, Chief Scientist, U.S. Government Accountability 
Office (GAO) 

Biography 200 

Mr. Todd C. Owen, Acting Deputy Assistant Commissioner, Office of Field 
Operations, U.S. Customs and Border Protection, Department of Homeland 
Security 

Oral Statement 200 

Written Statement 202 

Biography 206 

Dr. William K. Hagan, Acting Deputy Director, Domestic Nuclear Detection 
Office, Department of Homeland Security 

Oral Statement 207 

Written Statement 208 

Biography 211 

Discussion 

CBP Procedures After a Primary Alarm and the Effect of False Positives .... 212 

Steps Taken to Reduce False Positives and Negatives 213 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00004 Fmt5904 Sfmt5904 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



V 


Page 


Mission Critical Failure 215 

Energy Windowing to Improve PVT Performance 216 

The CBP Inspection System 217 

The Helium-3 Shortage and Potential Alternative Materials 218 

In What Circumstances Should ASPs Be Deployed? 219 

Expectations of a Cost-benefit Analysis on ASPs 220 

Metrics and Timelines for Making Decisions About ASPs 222 


Appendix: Answers to Post-Hearing Questions 

Dr. William K. Hagan, Acting Deputy Director, Domestic Nuclear Detection 
Office, Department of Homeland Security; and Mr. Todd C. Owen, Acting 
Deputy Assistant Commissioner, Office of Field Operations, U.S. Customs 

and Border Protection, Department of Homeland Security 230 

Mr. Gene Aloise, Director, Natural Resources and Environment, U.S. Govern- 
ment Accountability Office (GAO); Mr. Todd C. Owen, Acting Deputy As- 
sistant Commissioner, Office of Field Operations, U.S. Customs and Border 
Protection, Department of Homeland Security; and Dr. William K. Hagan, 
Acting Deputy Director, Domestic Nuclear Detection Office, Department 
of Homeland Security 235 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00005 Fmt5904 Sfmt5904 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00006 Fmt 5904 Sfmt 5904 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



THE SCIENCE OF SECURITY, PART I: LES- 
SONS LEARNED IN DEVELOPING, TESTING, 
AND OPERATING ADVANCED RADIATION 
MONITORS 


THURSDAY, JUNE 25, 2009 

House of Representatives, 
Subcommittee on Investigations and Oversight, 

Committee on Science and Technology, 

Washington, DC. 

The Subcommittee met, pursuant to call, at 10:00 a.m., in Room 
2318 of the Rayburn House Office Building, Hon. Brad Miller 
[Chairman of the Subcommittee] presiding. 
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HEARING CHARTER 

SUBCOMMITTEE ON INVESTIGATIONS AND OVERSIGHT 
COMMITTEE ON SCIENCE AND TECHNOLOGY 
U.S. HOUSE OF REPRESENTATIVES 

The Science of Security, Part I: Lessons 
Learned in Developing, Testing, and 
Operating Advanced Radiation Monitors 

THURSDAY, JUNE 25, 2009 

io:oo a.m.-12:oo p.m. 

2318 RAYBURN HOUSE OFFICE BUILDING 


Purpose 

The Subcommittee on Investigations and Oversight meets on June 25, 2009 to ex- 
amine problems with the Department of Homeland Security’s (DHS) efforts to ac- 
quire its next generation radiation monitors known as Advanced Spectroscopic Por- 
tals (ASPs). The ASP program has been under scrutiny since 2006 for failing to 
have clear-cut program requirements, an adequate test plan, sufficient timelines 
and development milestones or a transparent and comprehensive cost benefit anal- 
ysis. Since the Domestic Nuclear Detection Office (DNDO), a DHS component, was 
created in 2005, they have been responsible for researching, developing, testing and 
managing the program. 

The hearing will examine two new independent reports — one by the Government 
Accountability Office (GAO) and the other by the National Academy of Sciences — 
that identify ongoing and systematic problems in the testing and development of the 
ASP program. With an estimated program cost of $2-to-$3 billion the Subcommittee 
will evaluate the rigor of the overall test program, the technical abilities of the ASPs 
compared to existing radiation portal monitors and search for lessons from the ASP 
program that can be applied to future DHS acquisitions. 

Background 

Since the terrorist attacks of September 11, 2001, protecting the Nation from a 
nuclear or radiological attack has been a top national security priority. In 2002, to 
help address this potential threat, the U.S. Customs and Border Protection (CBP) 
agency began deploying radiation monitors at U.S. border sites and ports of entry 
so its officers could screen the more than 23 million containers of cargo that enter 
the country every year for radiological and nuclear materials. 

The equipment used to screen this cargo both then and now are polyvinyl toluene 
(PVT) or “plastic” portal monitors able to detect the presence of radioactive sources, 
but unable to identify the type of radiation present. The PVT monitors, while rel- 
atively inexpensive, robust and highly reliable, are unable to distinguish between 
radioactive sources that might be used to construct a nuclear bomb, such as Highly 
Enriched Uranium (HEU), and non-threatening naturally occurring radiological ma- 
terials (NORM) contained in ceramic tiles, zirconium sand or kitty liter, for in- 
stance. As a result, any time a PVT monitor detects a radioactive source the cargo 
is sent to “secondary” screening where CBP agents verify the detection of the radio- 
active source with a second PVT monitor and use hand-held Radioactive Isotope 
Identification Devices called RIIDs to help identify the source of radiation. 

This method of operation leads to many “secondary” inspections for naturally oc- 
curring radioactive material or radioactive material intended for benign purposes, 
such as radioactive medical isotopes. At the Los Angeles/Long Beach port of entry, 
for instance, PVT monitors routinely send up to 600 conveyances of cargo to sec- 
ondary inspection each day. The RIIDs, used in secondary inspections however, are 
limited in their abilities to locate and identify potential threat material in large 
cargo containers. As a result, CBP officers can consult with scientists in CBP’s Lab- 
oratories and Scientific Services (LSS) unit who can often help them enhance the 
ability to correctly identify the radioactive material of concern. As a last resort, CBP 
officers may physically search a cargo container by emptying its contents and closely 
scrutinizing it for potentially dangerous radioactive material. 
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If terrorists were to try to smuggle nuclear or radiological materials in container- 
ized cargo — and there are ample other pathways for such smuggling — they would 
likely try to shield or “mask” those materials in an attempt to make it more difficult 
to detect, identify and locate the material of concern. Shielding requires that lead 
or steal or other types of metal enclose the radioisotopes to hide its radioactive sig- 
nature. Potential terrorists may also attempt to “mask” threatening radioactive ma- 
terial by placing it together with or alongside other non-threatening material that 
has a natural radioactive signature, such as ceramic material, kitty litter or even 
bananas. Most nuclear security experts believe smuggled radioactive or nuclear ma- 
terial would be both shielded and masked in order to conceal it from being located 
and properly identified. Obviously, these efforts would make it harder to detect. 

In order to help both improve the flow of commerce by eliminating many of the 
false alarms that send cargo for secondary screening and to more accurately identify 
radioactive or nuclear material, the Department of Homeland Security (DHS) began 
developing Advanced Spectroscopic Portals (ASPs) in 2004. The ASPs were intended 
to both detect and identify radioactive material. In April 2005, the Domestic Nuclear 
Detection Office was created by National Security Presidential Directive-43/Home- 
land Security Presidential Directive-14 to, among other things, research, develop, 
test and acquire radiation detection equipment to be used by CBP and other federal 
agencies. The office was not formally established until October 2006 under the 
SAFE Port Act. 

From the very start of the ASP program, DNDO seemed to push for acquisition 
decisions well before the technology had demonstrated that it could live up to its 
promise. On July 14, 2006, Secretary of Homeland Security Michael Chertoff and 
the Director of DNDO, Vayl Oxford, announced contract awards to three companies 
worth an estimated $1.2 billion to develop the ASPs, including the Raytheon Com- 
pany, from Massachusetts, the Thermo Electron Company from Santa Fe, New Mex- 
ico and Canberra Industries from Connecticut. Both Chertoff and Oxford held a 
press conference to announce the billion dollar contract awards just a few months 
after highly critical reviews of the ASPs’ abilities by the GAO and the National In- 
stitute of Standards and Technology (NIST). 

In March 2006, GAO said: “it is not clear that the benefits of the new portals 
would be worth any increased cost to the program.” In June 2006, NIST submitted 
a report to DHS on results of side-by-side testing the previous year at the Nevada 
Test Site of both ASP and PVT systems. The DNDO had assumed that the ASPs 
would correctly identify HEU 95 percent of the time for both bare or unmasked 
HEU and HEU masked in a container with more benign radiological material. Yet, 
NIST found that the three best ASP systems tested identified HEU only 70 to 88 
percent of the time. Their ability to identify “masked” HEU was much worse. The 
three ASP manufacturers did this only 53 percent of the time (Raytheon), 45 percent 
of the time (Thermo) and 17 percent of the time (Canberra). “Despite these results,” 
the GAO found, “DNDO did not use the information from these tests in its cost-ben- 
efit analysis.” DNDO claimed that they assumed they would meet the 95 percent 
performance level at some point in the future but provided no data on why they 
reached this conclusion, said GAO. 

At the Chertoff-Oxford press conference in July 2006, then Secretary of Homeland 
Security, Michael Chertoff, said one of the key reasons for developing the ASPs and 
replacing the existing radiation monitors was to “have fewer false positives.” In Sep- 
tember 2007, Vayl Oxford, the Director of DNDO reiterated that point in testimony 
to Congress where he emphasized that the ASPs would reduce the number of false 
alarms from the nearly 600 experienced each day by the PVTs at the port of Long 
Beach in California, for instance, to 20 to 25 per day with the new ASP monitors. 

That was the hope, anyway. One of the criteria for ASP primary screening prior 
to certification of the new radiation monitors by the Secretary of Homeland Secu- 
rity, which is required by the appropriations committees, is that the ASPs must 
refer at least 80 percent fewer conveyances for further inspection than the PVTs. 
But in “field validation tests” earlier this year, by one of the two remaining contrac- 
tors, the ASPs being tested sent more innocent radioactive shipments to secondary 
screening than the older PVT monitors. The cause of the high false positives was 
apparently due to a software glitch. This was a serious concern to the Customs and 
Border Protection personnel who will have to operate and maintain the ASPs if and 
when they are certified and deployed. The contractor has reportedly corrected the 
software issue and intends to return the ASPs to field validation testing next 
month. 

Last fall, “integration” testing of the ASPs by the second remaining contractor 
was halted because of different technical troubles with its own software. The con- 
tractor corrected the problem and its ASP machines re-entered integration testing 
late last year. The contractor hopes to finish integration testing and begin field vali- 
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dation testing in early August. Still, both contractors are now many months behind 
schedule because technical issues have forced delays. 

Virtually any high-technology research and development program experiences 
bumps in the road, technical troubles and occasional set-backs. However, well man- 
aged programs have clear technical requirements and strategic goals. They ensure 
that the new technology being developed is thoroughly tested and adequately inte- 
grated into the operational plans and procedures of those who must operate them 
in the field. When these vital components are short changed, when the test plan is 
insufficient and the program’s research, development and testing methods are 
marred by scanty scientific rigor, the technical tools being developed are bound to 
suffer as a result. Cutting critical corners in the development process serves no one’s 
interests. Yet, from the start many of the leaders of the ASP program at DHS 
seemed more interested in fielding this technology then in vigilantly validating its 
performance and effectiveness. At the July 2006 press conference unveiling the con- 
tractors on the ASP program, for instance, Vayl Oxford said: “the priority for the 
first year ... is to get units out immediately.” Three years later, none of these 
units have yet cleared field validation tests. 

The policy governing the ASP program and the disproportionate focus on getting 
the ASP units into the field quickly never matched the multiple independent tech- 
nical assessments of the technology being developed and tested. Over the past three 
years the Government Accountability Office has issued six reports on the ASP pro- 
gram and testified before Congress multiple times on this matter. Last year the 
Homeland Security Institute, a Federally Funded Research and Development Center 
for DHS, issued a report on the ASPs that also criticized the ASP test program, say- 
ing it provided insufficient data. The National Academy’s of Science, which will re- 
lease an interim report on the ASPs that they have just concluded this week, will 
provide testimony at the Subcommittee hearing that echoes many of the concerns 
raised by GAO over the years. 

History of Problems 

In 2006, the GAO issued a harsh critique of the DNDO’s cost-benefit-analysis 
(CBA) of the ASPs. The DNDO analysis omitted critical test data that identified 
major technical problems with the ASPs and they drastically increased the procure- 
ment costs of the PVTs. In short, the GAO found DNDO’s cost-benefit analysis was 
“incomplete,” based on “unreliable” data and used “inflated cost estimates for PVT 
equipment.” 

In 2007, GAO concluded that tests of the ASPs conducted by DNDO were “biased” 
and “were not an objective and rigorous assessment of the ASPs’ capabilities.” The 
tests, for instance, used insufficient amounts of materials likely to mask or shield 
radioactive threat sources that terrorists might attempt to smuggle into the country. 
The tests, said GAO, did not attempt to test the limitations of the ASPs and “did 
not objectively test the performance” of currently used hand-held radiation detectors 
or RIIDs. 

Last year, in their own independent cost estimate of the ASP program, GAO 
found that the ASPs could cost about $3.1 billion, $1 billion more than the DNDO’s 
estimate. The GAO also found that the DNDO had often changed its deployment 
strategy, eliminating plans to develop ASP portals for rail, airport and seaport cargo 
screening terminals, for instance. As a result, GAO estimated the newest scaled 
back plan reduced the potential costs of the program to about $2 billion from 2008 
to 2017. The only documentation that DNDO provided to GAO for this major change 
in the ASP program was a one-page spread-sheet and DNDO has still not released 
an updated cost-benefit analysis of the program. 

In addition, GAO criticized DNDO’s decision not to complete computerized simula- 
tions or “injection studies” of the ASPs prior to certification by the Secretary of 
Homeland Security. The National Academy of Sciences has also found that computer 
modeling is critically important to the ASP program since running every potential 
radioactive smuggling scenario in live tests is unrealistic. Computer simulations 
would help provide a clearer assessment of the potential performance of the ASPs 
in actual smuggling incidents and effectiveness at identifying threatening radio- 
active material. DNDO, however, does not plan to complete the studies prior to the 
Secretary of Homeland Security’s decision on certification, which DNDO expects to 
occur in October. 

Problems Remain 

While DNDO’s past tests have been characterized as being unsound, incomplete 
and limited in scope, the GAO’s most recent work on the ASP program does point 
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to some improvements in the integrity of the latest round of tests. However, they 
also pinpointed significant technical limitations which have not yet been resolved. 

The ASP portals did prove more effective than the PVTs in detecting HEU mate- 
rials concealed by “light shielding.” However, differences between the ASPs and 
PVTs became less notable when shielding was slightly increased or decreased. In 
past tests there was virtually no difference in the performance of the two machines 
with regard to detecting other kinds of radioactive isotopes, such as those used for 
medical or industrial purposes, according to the GAO, except in one case where the 
ASPs performed worse than the PVTs. Whether these other forms of radioactive 
sources are sensed by a PVT or ASP machine they all require secondary inspection 
to determine why a payload contains radioactive material. In detecting HEU, the 
ASPs performed better only in one narrowly defined scenario, which many experts 
see as an unrealistic portrayal of a true attempted nuclear smuggling incident. None 
of the tests run by DNDO, for instance, included scenarios that utilized both “shield- 
ing” and “masking” as a means of attempting to smuggle radioactive or nuclear ma- 
terial. 

In addition, GAO and others have faulted DNDO for not focusing enough on at- 
tempting to improve the current radiation portal monitor program. Instead, DNDO 
has been nearly single-mindedly focused on developing Advanced Spectroscopic Por- 
tals at the expense of other far simpler alternatives. Surprisingly, for instance, 
DNDO has not completed efforts to improve the performance of PVTs by a method 
called Energy Windowing that could provide them with some limited, but enhanced, 
performance. Energy Windowing efforts are controversial and are believed to only 
provide modest enhancements to the performance of PVTs. But both GAO and CBP 
has been pushing DNDO to do more on this front for years. In addition, DNDO has 
not made efforts to upgrade the software in the hand-held radiation detection units 
known as RIIDs that could also provide a far less expensive alternative to enhanc- 
ing the operational effectiveness of radiation monitors. 

Because both remaining ASP contractors suffered from serious technical problems 
in their last round of testing, Customs and Border Protection (CBP) agency per- 
sonnel fear that if the ASPs are certified, procured and deployed that they will en- 
counter many problems in the field that will negatively impact their day-to-day op- 
erations and perhaps the technical effectiveness of the current radiation monitoring 
program to actually detect illicit nuclear or radiological material coming into the 
country. The GAO, National Academy of Sciences and others have also criticized 
DNDO for not seeking input from CBP officials on the ASP program from the start. 
The relationship has improved and DNDO does attempt to include CBP in critical 
decisions regarding the ASP program today. But many critics say perhaps one of 
DNDO’s biggest failures was the fact that they did not do this from the beginning, 
seeking input from the operational users of the technology that DNDO was tasked 
to research, test and develop. 

As a result of all of these issues, the ASPs continue to suffer from key questions 
about their ability to provide significant improved performance over existing radi- 
ation detection equipment currently fielded at U.S. ports. The Department of Home- 
land Security has already spent more than $235 million on the ASP program. But 
if the Secretary of Homeland Security certifies that the ASP monitors are worth in- 
vesting in this fall — -just three to four months from now — then $2 billion more may 
be invested to procure ASP radiation monitors. Yet, given the continued criticism 
of the narrowly focused and inadequate ASP test program, the limited technical im- 
provements they may offer over current radiation monitors and significant increased 
costs to maintain and operate the ASPs compared to the PVTs, the success of the 
program remains in doubt. 

Key Issues 

• Go Slow. Uncovering and resolving technical problems once newly developed 
radiation monitors are fielded may hinder the ability to detect and identify 
radioactive or nuclear material that poses a potential threat. It could disrupt 
operations at U.S. borders and ports of entry curtailing the flow of commerce 
and it will cost more to rectify these problems in the field, rather than in the 
laboratory or at the test range. Yet, rather than carefully testing and vali- 
dating the performance and effectiveness of the ASP monitors before a major 
procurement decision is made DHS has continually sought to get the ASPs 
into the field in spite of critical technical flaws identified during testing. 

• Cost Benefit Analysis. Even if the technical abilities of the ASPs are prov- 
en, their relative technical capabilities and increased costs must be carefully 
weighed in comparison to the existing radiation monitoring system in place 
today. Replacing a proven, less-costly system that has the confidence of its op- 
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erators, must be given careful consideration. The DNDO has not yet provided 
an updated cost-benefit-analysis that would validate a decision to procure the 
multi-billion dollar ASP equipment. 

• Judging Performance. As the House Committee on Appropriations has said 
in the past, procurement of the Advanced Spectroscopic Portal monitors 
should not proceed until they are deemed to add a “significant increase in 
operational effectiveness” over the current PVT system already in place. Last 
July, CBP, DNDO and the DHS management directorate jointly issued cri- 
teria for determining this increase in effectiveness in both “primary” and “sec- 
ondary” screening. In primary screening the criteria requires ASPs to detect 
potential threats as well as or better than PVTs, show improved detection of 
Highly Enriched Uranium and reduce innocent alarms. In secondary screen- 
ing the criteria requires ASPs to reduce the probability of misidentifying spe- 
cial nuclear material (HEU or plutonium) and reduce the average time to con- 
duct secondary screenings. The Secretary of Homeland Security must certify 
to Congress that the ASPs have met these criteria before funding for full- 
scale procurement of the ASPs goes forward. However, the criteria to measure 
this improvement are weak and rather vague. 

• Lessons Learned. The Department of Homeland Security must make great- 
er efforts to avoid rushing to acquisition decisions when the R&D is incom- 
plete. With ASPs, the research and development program itself has been hin- 
dered by a lack of rigorous scientific evaluations, and undemanding testing 
protocols. Moving to acquire systems plagued by such problems may endanger 
security and significantly increase the costs of the program. A review of 
DHS’s major programs by GAO last November found that 45 of 48 major pro- 
grams did not adhere to the agency’s own investment review process that 
helps provide appropriate oversight to address cost, schedule and performance 
problems. In FY 2008, the review found, DHS spent $147.5 million on the 
ASP program despite the fact it did not have a mission needs statement. The 
program also lacked operational requirements documents and an acquisition 
program baseline. 

Witnesses 

Panel I: 

Mr. Gene Aloise, Director, Natural Resources and Environment, Government Ac- 
countability Office 

Dr. Micah Lowenthal, Division on Earth and Life Studies, Nuclear and Radiation 
Studies Board, National Research Council, The National Academy of Sciences 

Panel II: 

Dr. William Hagan, Acting Deputy Director, Domestic Nuclear Detection Office 
(DNDO), Department of Homeland Security (DHS) 

Mr. Todd C. Owen, Acting Deputy Assistant Commissioner, Office of Field Oper- 
ations, U.S. Customs & Border Protection (CBP), Department of Homeland Security 
(DHS). 
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Chairman Miller. The hearing will now come to order. Good 
morning. Welcome to today’s hearing, The Science of Security: Les- 
sons Learned in Developing, Testing, and Operating Advanced Ra- 
diation Monitors. 

In the wake of the terrorist attacks on September 11, 2001, pre- 
venting the detonation of a nuclear or radiologic device — a ‘dirty 
bomb’ — in the United States has become a top national security ob- 
jective. We have invested billions of dollars since 9/11 to develop 
the means to prevent, detect and respond to any attack by weapons 
of mass destruction. We developed radiation monitors at our port 
and border crossings to screen millions of cargo containers entering 
the United States every year, hunting for radiological material that 
could be used for terrorist purposes. Since 2004, the Department of 
Homeland Security has spent more than $230 million on a program 
to develop a new radiation detection system called an Advanced 
Spectroscopic Portal, or ASP, that can both detect and identify nu- 
clear material. 

Congress expects that the funding federal agencies receive will 
be well spent. When it comes to scientifically challenging or tech- 
nically demanding programs, it pays to have well-prepared pro- 
gram requirements, demanding testing protocols and an inde- 
pendent and comprehensive cost-benefit analysis. Those vital steps, 
those vital program components help managers make informed de- 
cisions about whether to move forward with a technology develop- 
ment program or to replace a proven technology with a new tech- 
nology. Unfortunately, despite recent progress, the ASP program 
has suffered because it lacked all the preparatory steps of a well- 
managed program. We will hear about some of those problems 
today from the Government Accountability Office and the National 
Academies of Science. 

Over the years, the GAO has released six reports on the ASP 
program. The GAO found that some of the Domestic Nuclear Detec- 
tion Office’s tests were biased and did not provide a rigorous as- 
sessment of ASP’s capabilities. The Agency relied on incomplete 
and unreliable data in their cost-benefit analysis, omitting critical 
test data, inflating the cost of current radiation detectors and un- 
derestimating the cost of ASP monitors. The Department failed to 
produce a requirements document or adequate documentation re- 
garding major changes to the planned ASP deployment strategy. 
DNDO never considered the option of investigating improvements 
of the existing radiation portal monitoring program, both the PVT 
monitors and the hand-held detectors that Customs and Border 
Protection agents rely upon. 

The National Academy of Science’s interim report on the ASP 
program, released yesterday, reflects many of the same concerns. 
The Academy calls for significant restructuring of DHS testing pro- 
cedures for the ASP program. They question the criteria being used 
to judge the ASPs’ performance and they recommend that DHS not 
proceed with further ASP procurement until they address all the 
findings and recommendations in their report. 

GAO’s reports have provided a regular accounting of how the 
ASP program was going wrong. The Academy report provides a 
roadmap to how the program can be put back on track, assuming 
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that the Department determines that it is worth the cost and ef- 
fort. 

Radioactive materials have become a normal part of commerce. 
They are used for medical procedures and industrial applications. 
Bananas have radiation. Technology can help us detect and iden- 
tify radioactive sources in cargo containers but no technology can 
sort out good radioactive material intended for legitimate purposes 
from the bad radioactive materials intended to do us harm. As a 
result, human operators will still need to make important decisions 
often informed by intelligence efforts to keep the Nation secure. 
Well-trained, well-equipped people, law enforcement officers and 
Customs and Border Protection inspectors will always be critical to 
the equation. 

This hearing addresses our responsibility to the technological 
part of that equation. Before we move forward with a $2 billion or 
$3 billion program, we must ensure that we get our money’s worth 
from the new technology. Put another way, if we have $2 billion 
or $3 billion to spend to enhance our security, is this technology 
really how we should spend it? 

I look forward to hearing from all of our witnesses today. 

And now I recognize Dr. Broun, the Ranking Member from Geor- 
gia, for an opening statement. 

[The prepared statement of Chairman Miller follows:] 

Prepared Statement of Chairman Brad Miller 

In the wake of the terrorist attacks of September 11, 2001, preventing the detona- 
tion of a nuclear or radiological device in the U.S. has become a top national secu- 
rity objective. 

We have invested billions of dollars since 9/11 to develop the means to prevent, 
detect and respond to any attack by weapons of mass destruction. We have deployed 
radiation monitors at our ports and border crossings to screen millions of cargo con- 
tainers entering the U.S. every year, hunting for radiological material that could be 
used for terrorist purposes. Since 2004, the Department of Homeland Security 
(DHS) has spent more than $230 million on a program to develop a new radiation 
detection system called an Advanced Spectroscopic Portal or ASP that can both de- 
tect and identify nuclear material. 

Congress expects that the funding federal agencies receive will be well spent. 
When it comes to scientifically challenging or technically demanding programs, 
studies have shown that it pays to have well-prepared program requirements, de- 
manding testing protocols and an independent and comprehensive cost benefit anal- 
ysis. These vital program components help managers make informed decisions about 
whether to move forward with a technology development program, or to replace a 
proven technology with a new technology. Unfortunately, despite some recent 
progress, the ASP program has suffered because it lacked all the preparatory steps 
of a well managed program. We will hear about some of those problems today from 
both Government Accountability Office (GAO) and the National Academies of 
Science. 

Over the past three years the GAO has released six reports on the ASP program. 
The GAO found that some of the Domestic Nuclear Detection Office’s (DNDO) tests 
were “biased” and did not provide a rigorous assessment of the ASP’s capabilities. 
The agency relied on “incomplete” and “unreliable” data in their cost-benefit anal- 
ysis — omitting critical test data, inflating the costs of the current radiation detec- 
tors, and underestimating the costs of ASP monitors. The Department failed to 
produce a requirements document or adequate documentation regarding major 
changes to their planned ASP deployment strategy. DNDO never considered the op- 
tion of investing in improvements to the existing radiation portal monitoring pro- 
gram — both the current polyvinyl toluene (PVT) monitors and the hand-held detec- 
tors Customs and Border Protection agents rely upon. 

The National Academy of Sciences’ interim report on the ASP program, released 
yesterday, reflects many of the same concerns. The Academy calls for a significant 
restructuring of DHS testing procedures for the ASP program, they question the cri- 
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teria being used to judge the ASP’s performance and they recommend that DHS not 
proceed with further ASP procurement until they address all of the findings and 
recommendations in their report. 

GAO’s reports have provided a regular accounting of how the ASP program was 
going wrong; the Academy’s report provides a roadmap to how the program could 
be put back on track — assuming that the Department determines that is worth the 
cost and effort. 

Radioactive materials have become a normal part of commerce. They are used for 
medical procedures and industrial applications. Technology can help us detect and 
identify radioactive sources in cargo containers, but no technology can sort out 
“good” radioactive material intended for legitimate purposes from the “bad” radio- 
active material intended to do us harm. As a result, human operators will need to 
make important decisions, often informed by intelligence efforts, to keep the Nation 
secure. Well trained, well equipped people — law enforcement officers and customs 
and border protection inspectors — will always be critical to the equation. 

This hearing addresses our responsibility to the technological part of that equa- 
tion. Before we move forward with a two to three billion dollar program we must 
ensure that we get our money’s worth from the new technology. Put another way, 
if we have two or three billion dollars to spend to enhance our security, is this tech- 
nology really how we should spend it? 

I look forward to hearing from all of our witnesses today. 

Mr. Broun. Thank you, Mr. Chairman. I want to welcome our 
witnesses here today and to thank you for participating in this im- 
portant hearing on the Department of Homeland Security’s Ad- 
vanced Spectroscopic Portal program. I know it is hard to say, par- 
ticularly for the non-scientist, but I am a scientist — I am a physi- 
cian. 

Yesterday the House took up consideration of the DHS appro- 
priations bill, and later today the Science Committee’s Technology 
and Innovation Subcommittee, which I am also a Member of, will 
hold a hearing on cyber security. I also sit on the Homeland Secu- 
rity Committee’s Subcommittee on Emerging Threats, Cyber Secu- 
rity, Science and Technology as well. Needless to say, DHS has 
kept me busy this week. 

This morning we will look into the status of the Department’s on- 
going development of the next -generation radiation portal monitors, 
get an update from GAO on their continued work, and receive a re- 
port from the National Research Council. It goes without saying 
that this program has been followed closely for some time now and 
thankfully many of the testing issues that GAO has brought up in 
previous reports seem to have been mitigated, at least somewhat. 
However, this program is far from out of the woods. In their most 
recent analysis, GAO and the Academy raised new issues relating 
to the rigor of the testing and the certification process and offer 
paths forward for potential acquisition in the future. I hope DHS 
takes these recommendations seriously, and I look forward to en- 
suring that they are not summarily dismissed for the sake of arbi- 
trary timetables. We see that frequently here in government. 

Looking forward, DHS should conduct a rigorous cost-benefit 
analysis that takes into account updated threat assessments, a re- 
view of all variations of concepts of operations, potential upgrades 
to existing technologies and independent cost estimates. It also 
needs to weigh the pros and cons not just of ASP versus Polyvinyl 
Toluene (PVT) and Radio-Isotope Identification Devices (RIID), but 
also whether the additional capability gained outweighs the needs 
of other aspects of the global nuclear detection architecture. Unfor- 
tunately, this may be hard to do at this point considering GAO in- 
dicated earlier this year that the Domestic Nuclear Detection Office 
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had, to quote GAO, “not developed an overarching strategic plan to 
guide its development of a more comprehensive global strategy for 
nuclear detection.” 

All of these factors need to be taken into consideration as DHS 
moves toward an acquisition. Even then I will remain cautious, 
given the Department’s track record with past acquisitions. Many 
of the issues we are dealing with today could have been prevented 
by engaging the end-users early on in the process and also by clear- 
ly defining the requirements and simply following existing Depart- 
ment acquisition processes. 

Last week this subcommittee held a hearing on issues plaguing 
NPOESS. The AST program exhibits eerie similarities to that pro- 
gram in that it attempted to link research and development activi- 
ties with the acquisition of an operational system and had unclear 
architectural priorities. Let us hope other federal programs can 
learn from these lessons and protect taxpayers from future ineffi- 
ciencies and waste. 

Mr. Chairman, I yield back the balance of my time and I look 
forward to our witnesses’ testimony. Thank you. 

[The prepared statement of Mr. Broun follows:] 

Prepared Statement of Representative Paul C. Broun 

Thank you, Mr. Chairman. I want to welcome the witnesses here today, and 
thank them for participating in this important hearing on the Department of Home- 
land Security’s (DHS) Advanced Spectroscopic Portal (ASP) program. 

Yesterday the House took up consideration of the DHS Appropriation bill, and 
later today the Science Committee’s Technology and Innovation Subcommittee, 
which I also sit on, will hold a hearing on cyber security. I also sit on the Homeland 
Security Committee’s Subcommittee on Emerging Threats, Cyber Security, Science 
and Technology, as well. Needless to say, DHS has kept me busy this week. 

This morning we will look into the status of the Department’s ongoing develop- 
ment of next-generation Radiation Portal Monitors, get an update from General Ac- 
countability Office (GAO) on their continuing work, and receive a report from the 
National Academy of Sciences. It goes without saying that this program has been 
followed closely for some time now, and thankfully many of the testing issues that 
GAO brought up in previous reports seem to be mitigated. However, this program 
is far from “out of the woods.” In their most recent analysis, GAO and the Academy 
raise new issues relating to the rigor of the testing and certification process, and 
offer paths forward for a potential acquisition in the future. I hope DHS takes these 
recommendations seriously, and I look forward to ensuring that they are not sum- 
marily dismissed for the sake of arbitrary timetables. 

Looking forward, DHS should conduct a rigorous Cost-Benefit Analysis (CBA) that 
takes into account updated threat assessments, a review of all variations of Con- 
cepts of Operations (CONOPS), potential upgrades for existing technologies, and 
independent cost estimates. It also needs to weigh the pros and cons of not just ASP 
versus Polyvinyl Toluene (PVT) and Radio-isotope Identification Devices (RIID), but 
also whether the additional capability gained outweighs the needs of other aspects 
of the Global Nuclear Detection Architecture. Unfortunately, this may be hard to 
do at this point considering GAO indicated earlier this year that the Domestic Nu- 
clear Detection Office (DNDO) had “not developed an overarching strategic plan to 
guide its development of a more comprehensive global strategy for nuclear detec- 
tion.” 

All of these factors need to be taken into consideration as DHS moves toward an 
acquisition. Even then, I will remain cautious given the Department’s track record 
with past acquisitions. Many of the issues we are dealing with today could have 
been prevented by engaging the end-users early in the process, clearly defining re- 
quirements, and simply following existing Department acquisition processes. 

Last week this subcommittee held a hearing on issues plaguing the National 
Polar-Orbiting Environmental Satellite System (NPOESS). The ASP program exhib- 
its eerie similarities to that program in that it attempted to link research and devel- 
opment activities with the acquisition of an operational system, and had unclear ar- 
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chitectural priorities. Let’s hope other federal programs can learn from these lessons 
and protect taxpayers from future inefficiencies and waste. 

With that, Mr. Chairman, I yield back my time. 

Thank you. 

Chairman Miller. Thank you, Dr. Broun. 

Panel I: 

Chairman Miller. I am now pleased to introduce our panel of 
witnesses. Mr. Gene Aloise is the Director of Natural Resources 
and Environment at the Government Accountability Office, GAO, 
and Dr. Micah Lowenthal is the Director of the Nuclear Security 
and Nuclear Facility Safety Program in the Nuclear and Radiation 
Studies Board at the National Research Council of the National 
Academy of Sciences. Is that how you describe your job at a cock- 
tail party? 

As our witnesses should know, you each have five minutes for 
your spoken testimony. Your written testimony will be included in 
the record for the hearing. When you have all completed your spo- 
ken testimony, we will begin with questions and each Member will 
have five minutes to question the panel. It is the practice of this 
subcommittee to receive testimony under oath. Do either of you 
have any objection to taking an oath? Okay. You also have the 
right to be represented by counsel. Do either of you have counsel 
here? If you would please stand and raise your right hand? Do you 
swear to tell the truth and nothing but the truth? 

The record will reflect that both Mr. Aloise and Dr. Lowenthal 
took the oath. Mr. Aloise, you may begin. 

STATEMENT OF MR. GENE ALOISE, DIRECTOR, NATURAL RE- 
SOURCES AND ENVIRONMENT, U.S. GOVERNMENT ACCOUNT- 
ABILITY OFFICE (GAO) 

Mr. Aloise. Thank you, Mr. Chairman. 

Mr. Chairman and Members of the Subcommittee, I am pleased 
to be here today to discuss DHS’s plans to develop and test ad- 
vanced portal monitors for use at the Nation’s borders to prevent 
nuclear materials from being smuggled into the United States. Ac- 
cording to DHS, the current system of radiation detection equip- 
ment is effective and does not impede the flow of commerce. How- 
ever, DHS wants to improve the capabilities of the existing equip- 
ment with new equipment. 

One of the major drawbacks of the new equipment is the sub- 
stantially higher cost compared to the existing equipment. We esti- 
mated in September 2008 that the life cycle cost of each standard 
cargo version of the ASP to be about $823,000 compared to about 
$308,000 for the PVT standard cargo portal and that the total pro- 
gram cost for the Domestic Nuclear Detection Office’s (DNDO) lat- 
est deployment plan would be about $2 billion. 

Since 2006, we have issued six reports, and including today, four 
testimonies, and have made 19 recommendations for improving 
DNDO’s efforts to develop and test portal monitors. Our concerns 
have focused on the need for realistic and objective testing of ASPs, 
full disclosure and reporting of testing limitations, development of 
cost estimates that consider the full cost to deploy the new equip- 
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ment, and development of a sound cost-benefit analysis which de- 
termines whether the marginal increase in security from the ASPs 
is worth its very high cost. 

My testimony today is based on our recent report, which assessed 
the most recent round of ASP testing. I will also discuss the lessons 
learned from ASP testing. Our work on the latest round of ASP 
testing found that DHS increased the rigor in comparison with pre- 
vious tests, and thereby added credibility to the test results. How- 
ever, we still question whether the benefits of the ASP justify the 
high cost. In particular, DHS’s criteria for significant increase in 
operational effectiveness require only marginal improvement in the 
detection of certain weapons-usable nuclear material. The marginal 
improvement required of ASPs is particularly notable given that 
DNDO has not completed efforts to enhance the performance of the 
current generation of equipment. Customs and Border Protection 
(CBP) officials have told us that they have repeatedly urged DNDO 
to investigate improving the performance of the equipment through 
what is known as energy windowing. DNDO has collected the data 
necessary to do this, but has not yet completed efforts to analyze 
the data and further improve the technique. 

Our analysis of the new test results shows that ASPs detected 
certain nuclear materials better than PVTs when shielding ap- 
proximated DOE threat guidance, which is based on light shield- 
ing. However, differences between the two systems were hard to 
recognize when shielding was slightly increased or decreased. Both 
systems had difficulty in detecting nuclear materials when shield- 
ing was somewhat greater than threat guidance, which is set to 
match the extent of the detection limits of the PVTs. Importantly, 
the threat guidance is not a realistic approximation of how a ter- 
rorist might shield nuclear material to successfully smuggle it un- 
detected through a border crossing. 

In addition, DNDO underestimated the time needed for ASP test- 
ing, which was originally supposed to be finished by September 
2008. DHS’s most recent schedule anticipated ASP certification 
around May 2009, but testing has been delayed even further, and 
DHS has not updated its schedule. As far as we know, certification 
is scheduled for some time this fall. 

Of concern to us is the fact that DNDO does not plan to complete 
simulations that could provide additional insights into ASP capa- 
bilities and limitations prior to certification. On this point, DNDO 
does not seem to have learned from past mistakes. A rush to deploy 
ASPs before all testing is complete leads to shortcuts in testing and 
raises questions among stakeholders about the equipment’s effec- 
tiveness and reliability. Furthermore, DNDO has not yet updated 
its cost-benefit analysis, which might show that DNDO’s plan to re- 
place existing equipment is not justified. 

One of the primary lessons to be learned from our work is to 
avoid the pitfalls in testing that stem from a rush to procure new 
technologies. In the case of the ASP, a push to rush to replace ex- 
isting equipment led to a testing program which lacked scientific 
rigor. Even for the new round of testing, DNDO consistently under- 
estimated the time necessary to conduct tests and resolve prob- 
lems, and testing is still not completed. 
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In closing, we believe that given the importance of this new 
equipment to our national security, that Congress and the Amer- 
ican taxpayer still need to know three things: does the equipment 
work, how much will it cost; and does the marginal increase in se- 
curity justify its very high cost. 

Mr. Chairman, that concludes my remarks and I would be happy 
to respond to any questions you and the Ranking Member may 
have. 

[The prepared statement of Mr. Aloise follows:] 

Prepared Statement of Gene Aloise 

Combating Nuclear Smuggling: Lessons 
Learned From DHS Testing of Advanced 
Radiation Detection Portal Monitors 


Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to discuss GAO’s work on the Department of Home- 
land Security’s (DHS) testing of advanced spectroscopic portal (ASP) radiation detec- 
tion monitors. As you are aware, the national security mission of U.S. Customs and 
Border Protection (CBP), an agency within DHS, includes screening for smuggled 
nuclear or radiological material that could be used in a nuclear weapon or radio- 
logical dispersal device (a “dirty bomb”). To screen cargo at ports of entry, CBP con- 
ducts primary inspections with radiation detection equipment called portal mon- 
itors — large stationary detectors through which cargo containers and vehicles pass 
as they enter the United States. When radiation is detected, CBP conducts sec- 
ondary inspections using a second portal monitor to confirm the original alarm and 
a hand-held radioactive isotope identification device to identify the radiation’s 
source and determine whether it constitutes a threat. 

The polyvinyl toluene (PVT) portal monitors currently in use can detect radiation 
but cannot identify the type of material causing an alarm. As a result, the monitors’ 
radiation alarms can be set off even by benign, naturally occurring radioactive mate- 
rial. One way to reduce the rate of such innocent alarms — and thereby minimize un- 
necessary secondary inspections and enhance the flow of commerce — is to adjust the 
operational thresholds (i.e., operate the PVTs at a reduced level of sensitivity). How- 
ever, reducing the sensitivity may make it more difficult to detect certain nuclear 
materials. 

To address the limitations of current-generation portal monitors, DHS’s Domestic 
Nuclear Detection Office (DNDO) in 2005 began to develop and test ASPs, which 
are designed to both detect radiation and identify the source. 1 DNDO hopes to use 
the new portal monitors to replace at least some PVTs currently used for primary 
screening, as well as PVTs and hand-held identification devices currently used for 
secondary screening. However, in September 2008, we estimated the life cycle cost 
of each standard cargo version of the ASP (including deployment costs) to be about 
$822,000, compared with about $308,000 for the PVT standard cargo portal, and the 
total program cost for DNDO’s latest plan for deploying radiation portal monitors — 
which relies on a combination of ASPs and PVTs and does not deploy radiation por- 
tal monitors at all border crossings — to be about $2 billion. 2 

Concerned about the performance and cost of the ASP monitors, Congress re- 
quired the Secretary of Homeland Security to certify that the monitors will provide 
a “significant increase in operational effectiveness” before DNDO obligates funds for 
full-scale ASP procurement. 3 The Secretary must submit separate certifications for 


1 DNDO was established within DHS in 2005; its mission includes developing, testing, acquir- 
ing, and supporting the deployment of radiation detection equipment at U.S. ports of entry. CBP 
began deploying portal monitors in 2002, prior to DNDO’s creation, under the radiation portal 
monitor project. 

2 GAO, Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy Advanced Radi- 
ation Detection Portal Monitors Is Likely to Exceed the Department’s Previous Cost Estimates, 
GAO-08—1 108R (Washington, D.C.: Sept. 22, 2008). 

3 Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 2069 (2007); 
Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 2009, Pub. L. 
No. 110-329, 121 Stat. 3574, 3679 (2008). 
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primary and secondary inspection. In response, CBP, DNDO, and the DHS manage- 
ment directorate jointly issued criteria in July 2008 for determining whether the 
new technology provides a significant increase in operational effectiveness. The pri- 
mary screening criteria require that the new portal monitors detect potential 
threats as well as or better than PVTs, show improved performance in detection of 
highly enriched uranium (HEU), and reduce innocent alarms. To meet the sec- 
ondary screening criteria, the new portal monitors must reduce the probability of 
misidentifying special nuclear material (e.g., HEU and plutonium) and the average 
time to conduct secondary screenings. 

DNDO designed and coordinated a new series of tests, originally scheduled to run 
from April 2008 through September 2008, to determine whether the new portal 
monitors meet the certification criteria for primary and secondary screening and are 
ready for deployment. Key phases of this testing program include concurrent testing 
led by DNDO of the new and current equipment’s ability to detect and identify 
threats and of ASPs’ readiness to be integrated into operations for both primary and 
secondary screening at ports of entry; field validation led by CBP at four northern 
and southern border crossings and two seaports; and an independent evaluation, led 
by the DHS Science and Technology Directorate at one of the seaports, of the new 
portal monitors’ effectiveness and suitability. 

Since 2006, we have issued six reports and four testimonies on development of 
radiation detection portal monitors, including today’s testimony, and have made 19 
recommendations for improving DNDO’s efforts to develop and test portal monitors. 
Our concerns have focused on key areas in which DNDO’s efforts have lacked the 
necessary rigor given ASPs’ high cost and the importance of the radiation portal 
monitor project to our national security. These areas include objective and realistic 
testing of ASPs’ performance in comparison with the performance of current-genera- 
tion equipment; full disclosure and reporting of the limitations of tests used to sup- 
port a decision by the Secretary of Homeland Security on ASP certification; develop- 
ment of a cost estimate that considers the full costs of the plan for deploying radi- 
ation detection portal monitors; and development of a cost-benefit analysis based on 
ASPs’ demonstrated performance and a complete accounting of the portal monitor 
project’s costs. (App. I presents a summary of our key findings and recommenda- 
tions related to ASPs.) As I will discuss today, DNDO has improved the rigor of test- 
ing but has not yet updated the cost-benefit analysis that is critical to a decision 
on whether to replace radiation detection equipment already deployed at ports of 
entry with the significantly more expensive ASPs. 

Specifically, my testimony discusses ( 1 ) our key findings on the most recent round 
of ASP testing and (2) lessons from ASP testing that can be applied to other DHS 
technology investments. These findings are based on our report released this week 
and other related GAO reports. 4 We conducted this performance audit work in June 
2009 in accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, appro- 
priate evidence to produce a reasonable basis for our findings and conclusions based 
on our audit objectives. We believe that the evidence obtained provides a reasonable 
basis for our statement today. 

The Latest Round of Testing Highlights the Limitations of ASPs 

Our report on the latest round of ASP testing found that DHS increased the rigor 
of ASP testing in comparison with previous tests and that a particular area of im- 
provement was in the performance testing at the Nevada Test Site, where DNDO 
compared the capability of ASP and current-generation equipment to detect and 
identify nuclear and radiological materials. For example, unlike in prior tests, the 
plan for the 2008 performance test stipulated that there would be no system con- 
tractor involvement in test execution. Such improvements addressed concerns we 
previously raised about the potential for bias and provided credibility to the results. 

Nevertheless, based on the following factors, we continue to question whether the 
benefits of the new portal monitors justify the high cost: 

• The DHS criteria for a significant increase in operational effectiveness. Our 
chief concern with the criteria is that they require a marginal improvement 
over current-generation portal monitors in the detection of certain weapons- 
usable nuclear materials when ASPs are deployed for primary screening. 
DNDO considers detection of such materials to be a key limitation of current- 
generation portal monitors. We are particularly concerned about the marginal 


4 GAO, Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detec- 
tion Portal Monitors, but Preliminary Results Show Limits of the New Technology, GAO-09—655 
(Washington, D.C.: May 21, 2009). 
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improvement required of ASPs because the detection threshold for the cur- 
rent-generation portal monitors does not specify a level of radiation shielding 
that smugglers could realistically use. DOE and national laboratory officials 
told us that DOE’s threat guidance used to set the current detection threshold 
is based not on an analysis of the capabilities of potential smugglers to take 
effective shielding measures but rather on the limited sensitivity of PVTs to 
detect anything more than certain lightly shielded nuclear materials. DNDO 
officials acknowledge that both the new and current-generation portal mon- 
itors are capable of detecting certain nuclear materials only when unshielded 
or lightly shielded. The marginal improvement in detection of such materials 
required of ASPs is particularly notable given that DNDO has not completed 
efforts to fine-tune PVTs’ software and thereby improve sensitivity to nuclear 
materials. DNDO officials expect they can achieve small improvements in 
sensitivity, but DNDO has not yet funded efforts to fine-tune PVTs’ software. 
In contrast to the marginal improvement required in detection of certain nu- 
clear materials, the primary screening requirement to reduce the rate of inno- 
cent alarms could result in hundreds of fewer secondary screenings per day, 
thereby reducing CBP’s workload and delays to commerce. In addition, the 
secondary screening criteria, which require ASPs to reduce the probability of 
misidentifying special nuclear material by one-half, address the inability of 
relatively small hand-held devices to consistently locate and identify potential 
threats in large cargo containers. 

• Preliminary results of performance testing and field validation. The prelimi- 
nary results presented to us by DNDO are mixed, particularly in the capa- 
bility of ASPs used for primary screening to detect certain shielded nuclear 
materials. Preliminary results show that the new portal monitors detected 
certain nuclear materials better than PVTs when shielding approximated 
DOE threat guidance, which is based on light shielding. In contrast, dif- 
ferences in system performance were less notable when shielding was slightly 
increased or decreased: Both the PVTs and ASPs were frequently able to de- 
tect certain nuclear materials when shielding was below threat guidance, and 
both systems had difficulty detecting such materials when shielding was 
somewhat greater than threat guidance. With regard to secondary screening, 
ASPs performed better than hand-held devices in identification of threats 
when masked by naturally occurring radioactive material. However, dif- 
ferences in the ability to identify certain shielded nuclear materials depended 
on the level of shielding, with increasing levels appearing to reduce any ASP 
advantages over the hand-held identification devices. Other phases of testing 
uncovered multiple problems in meeting requirements for successfully inte- 
grating the new technology into operations at ports of entry. Of the two ASP 
vendors participating in the 2008 round of testing, one has fallen behind due 
to severe problems encountered during testing of ASPs’ readiness to be inte- 
grated into operations at ports of entry (“integration testing”); the problems 
may require that the vendor redo previous test phases to be considered for 
certification. The other vendor’s system completed integration testing, but 
CBP suspended field validation after two weeks because of serious perform- 
ance problems resulting in an overall increase in the number of referrals for 
secondary screening compared with existing equipment. 

• DNDO’s plans for computer simulations. DNDO does not plan to complete in- 
jection studies — computer simulations for testing the response of ASPs and 
PVTs to simulated threat objects concealed in cargo containers — prior to the 
Secretary of Homeland Security’s decision on certification even though delays 
to the ASP test schedule have allowed more time to conduct the studies. Ac- 
cording to DNDO officials, injection studies address the inability of perform- 
ance testing to replicate the wide variety of cargo coming into the United 
States and the inability to place special nuclear material and other threat ob- 
jects in cargo during field validation. DNDO had earlier indicated that injec- 
tion studies could provide information comparing the performance of the two 
systems as part of the certification process for both primary and secondary 
screening. However, DNDO subsequently decided that performance testing 
would provide sufficient information to support a decision on ASP certifi- 
cation. DNDO officials said they would instead use injection studies to sup- 
port effective deployment of the new portal monitors. 

• Lack of an updated cost-benefit analysis. DNDO has not yet updated its cost- 
benefit analysis to take into account the results of the latest round of ASP 
testing. An updated analysis that takes into account the results from the lat- 
est round of testing, including injection studies, might show that DNDO’s 
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plan to replace existing equipment with ASPs is not justified, particularly 
given the marginal improvement in detection of certain nuclear materials re- 
quired of ASPs and the potential to improve the current-generation portal 
monitors’ sensitivity to nuclear materials, most likely at a lower cost. DNDO 
officials said they are currently updating the ASP cost-benefit analysis and 
plan to complete it prior to a decision on certification by the Secretary of 
Homeland Security. 

Our report recommended that the Secretary of Homeland Security direct DNDO 
to (1) assess whether ASPs meet the criteria for a significant increase in operational 
effectiveness based on a valid comparison with PVTs’ full performance potential and 
(2) revise the schedule for ASP testing and certification to allow sufficient time for 
review and analysis of results from the final phases of testing and completion of all 
tests, including injection studies. We further recommended that, if ASPs are cer- 
tified, the Secretary direct DNDO to develop an initial deployment plan that allows 
CBP to uncover and resolve any additional problems not identified through testing 
before proceeding to full-scale deployment. DHS agreed to a phased deployment that 
should allow time to uncover ASP problems but disagreed with GAO’s other rec- 
ommendations, which we continue to believe remain valid. 

Procurement Decisions for New Technologies Require Rigorous Testing 
and Thorough Analysis of Results 

The challenges DNDO has faced in developing and testing ASPs illustrate the im- 
portance of following existing DHS policies as well as best practices for investments 
in complex homeland security acquisitions and for testing of new technologies. The 
DHS investment review process calls for executive decision-making at key points in 
an investment’s life cycle and includes many acquisition best practices that, if ap- 
plied consistently, could help increase the chances for successful outcomes. However, 
we reported in November 2008 that, for the period from fiscal year 2004 through 
the second quarter of fiscal year 2008, DHS had not effectively implemented or ad- 
hered to its investment review process due to a lack of senior management officials’ 
involvement as well as limited monitoring and resources. 5 In particular, of DHS’s 
48 major investments requiring milestone and annual reviews under tbe Depart- 
ment’s investment review policy, 45 were not assessed in accordance with this pol- 
icy. In addition, many major investments, including DNDO’s ASP program, had not 
met the Department’s requirements for basic acquisition documents necessary to in- 
form the investment review process. As a result, DHS had not consistently provided 
the oversight needed to identify and address cost, schedule, and performance prob- 
lems in its major investments. Among other things, our November 2008 report rec- 
ommended that the Secretary of Homeland Security direct component heads, such 
as the Director of DNDO, to ensure that the components have established processes 
to manage major investments consistent with departmental policies. DHS generally 
concurred with our recommendations, and we noted that DHS had begun several 
efforts to address shortcomings in the investment review process identified in our 
report, including issuing an interim directive requiring DHS components to align 
their internal policies and procedures by the end of the third quarter of fiscal year 
2009. In January 2009, DHS issued a memorandum instructing component heads 
to create acquisition executives in their organizations to be responsible for manage- 
ment and oversight of component acquisition processes. If fully implemented, these 
steps should help ensure that DHS components have established processes to man- 
age major investments. 

Based on our body of work on ASP testing, one of the primary lessons to be 
learned is to avoid the pitfalls in testing that stem from a rush to procure new tech- 
nologies. GAO has previously reported on the negative consequences of pressures 
imposed by closely linking testing and development programs with decisions to pro- 
cure and deploy new technologies, including the creation of incentives to postpone 
difficult tests and limit open communication about test results. 6 We found that test- 
ing programs designed to validate a product’s performance against increasing stand- 
ards for different stages in product development are a best practice for acquisition 
strategies for new technologies. In the case of ASPs, the push to replace existing 
equipment with the new portal monitors led to a testing program that until recently 
lacked the necessary rigor. Even for the most recent round of testing, DNDO’s 
schedule consistently underestimated the time required to conduct tests, resolve 


5 GAO, Department of Homeland Security: Billions Invested in Major Programs Lack Appro- 
priate Oversight, GAO-09-29 (Washington, D.C.: Nov. 18, 2008). 

6 GAO, Best Practices: A More Constructive Test Approach Is Key to Better Weapon System Out- 
comes, GAO/NSIAD-00— 199 (Washington, D.C.: July 31, 2000). 
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problems uncovered during testing, and complete key documents, including final 
test reports. In addition, DNDO’s original working schedule did not anticipate the 
time required to update its cost-benefit analysis to take into account the latest test 
results. The schedule anticipated completion of testing in mid-September 2008 and 
the DHS Secretary’s decision on ASP certification between September and Novem- 
ber 2008. However, testing is still not completed, and DNDO took months longer 
than anticipated to complete the final report on performance testing. 

As previously mentioned, a number of aspects of the latest round of ASP testing 
increased the rigor in comparison with earlier rounds and, if properly implemented, 
could improve the rigor in DHS’s testing of other advanced technologies. Key aspects 
included the following: 

• Criteria for ensuring test requirements are met. The test and evaluation mas- 
ter plan established criteria requiring that the ASPs meet certain require- 
ments before starting or completing any test phase. For example, the plan re- 
quired that ASPs have no critical or severe issues rendering them completely 
unusable or impairing their function. The criteria provided a formal means 
to ensure that ASPs met certain basic requirements prior to the start of each 
phase of testing. DNDO and CBP adhered to the criteria even though doing 
so resulted in integration testing taking longer than anticipated and delaying 
the start of field validation. 

• Participation of the technology end-user. The participation of CBP (the end- 
user of the new portal monitors) provided an independent check, within DHS, 
of DNDO’s efforts to develop and test the new portal monitors. For example, 
CBP added a final requirement to integration testing before proceeding to 
field validation to demonstrate ASPs’ ability to operate for 40 hours without 
additional problems and thereby provide for a productive field validation. In 
addition, the participation of CBP officers in the 2008 round of performance 
testing allowed DNDO to adhere more closely than in previous tests to CBP’s 
standard operating procedure for conducting a secondary inspection using the 
hand-held identification devices, thereby providing for an objective test. 

• Participation of an independent test authority. The DHS Science and Tech- 
nology Directorate, which is responsible for developing and implementing the 
Department’s test and evaluation policies and standards, will have the lead 
role in the final phase of ASP testing and thereby provide an additional inde- 
pendent check on testing efforts. The Science and Technology Directorate 
identified two critical questions, related to ASPs’ operational effectiveness 
(i.e., detection and identification of threats) and suitability (e.g., reliability, 
maintainability, and supportability), and drafted its own test plan to address 
those questions. 

Mr. Chairman, this completes my prepared statement. I would be happy to re- 
spond to any questions that you or other Members of the Subcommittee may have 
at this time. 
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Appendix I: 


Key Findings and Recommendations from 
Related GAO Products on Testing 
and Development of ASPs 

Combating Nuclear Smuggling: DHS Has Made Progress Deploying Radiation Detec- 
tion Equipment at U.S. Ports-of -Entry, But Concerns Remain. GAO-06-389. 
Washington, D.C.: March 22, 2006. 

• Key findings. Prototypes of advanced spectroscopic portals (ASP) were ex- 
pected to be significantly more expensive than current-generation portal mon- 
itors but had not been shown to be more effective. For example, Domestic Nu- 
clear Detection Office (DNDO) officials’ preliminary analysis of 10 ASPs test- 
ed at the Nevada Test Site found that the new portal monitors outperformed 
current-generation equipment in detecting numerous small, medium-size, and 
threat-like radioactive objects and were able to identify and dismiss most nat- 
urally occurring radioactive material. However, the detection capabilities of 
both types of portal monitors converged as the amount of source material de- 
creased. 

• Recommendations. We recommended that the Secretary of Homeland Security 
work with the Director of DNDO to analyze the benefits and costs of deploy- 
ing ASPs before any of the new equipment is purchased to determine whether 
any additional detection capability is worth the additional cost. We also rec- 
ommended that the total program cost estimate for the radiation portal mon- 
itor project be revised after completion of the cost-benefit analysis. 

Combating Nuclear Smuggling: DHS’s Cost-Benefit Analysis to Support the Purchase 
of New Radiation Detection Portal Monitors Was Not Based on Available Per- 
formance Data and Did Not Fully Evaluate All the Monitors’ Costs and Benefits. 
GAO-07-133R. Washington, D.C.: October 17, 2006. 

Combating Nuclear Smuggling: DHS’s Decision to Procure and Deploy the Next Gen- 
eration of Radiation Detection Equipment Is Not Supported by Its Cost-Benefit 
Analysis. GAO-07-581T. Washington, D.C.: March 14, 2007. 

• Key findings. DNDO’s cost-benefit analysis issued in response to our March 
2006 recommendation did not provide a sound analytical basis for DNDO’s de- 
cision to purchase and deploy ASPs. We identified a number of problems with 
the analysis of both the performance of the new portal monitors and the costs. 
With regard to performance, DNDO did not use the results of its own tests 
and instead relied on assumptions of the new technology’s anticipated per- 
formance level. In addition, the analysis focused on identifying highly en- 
riched uranium (HEU) and did not consider how well the new portal monitors 
can correctly detect or identify other dangerous radiological or nuclear mate- 
rials. With regard to costs, DNDO did not follow the DHS guidelines for per- 
forming cost-benefit analyses and used questionable assumptions about the 
procurement costs of portal monitor technology. 

• Recommendations. We recommended that DHS and DNDO conduct a new 
cost-benefit analysis using sound analytical methods, including actual per- 
formance data and a complete accounting of all major costs and benefits as 
required by DHS guidelines, and that DNDO conduct realistic testing for both 
ASPs and current-generation portal monitors. 

Combating Nuclear Smuggling: DNDO Has Not Yet Collected Most of the National 
Laboratories’ Test Results on Radiation Portal Monitors in Support of DNDO’s 
Testing and Development Program. GAO-07-347R. Washington, D.C.: March 9, 
2007. 

• Key findings. DNDO had not collected a comprehensive inventory of testing 
information on current-generation portal monitors. Such information, if col- 
lected and used, could improve DNDO’s understanding of how well portal 
monitors detect different radiological and nuclear materials under varying 
conditions. In turn, this understanding would assist DNDO’s future testing, 
development, deployment, and purchases of portal monitors. 

• Recommendations. We recommended that the Secretary of Homeland Secu- 
rity, working with the Director of DNDO, collect reports concerning all of the 
testing of current-generation portal monitors and review the test reports in 
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order to develop an information database on how the portal monitors perform 
in both laboratory and field tests on a variety of indicators, such as their abil- 
ity to detect specific radiological and nuclear materials. 

Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Test- 
ing of Next Generation Radiation Detection Equipment. GAO-07-1247T. Wash- 
ington, D.C.: September 18, 2007. 

• Key findings. We found that tests conducted by DNDO in early 2007 were not 
an objective and rigorous assessment of the ASPs’ capabilities. Specifically, 
we raised concerns about DNDO using biased test methods that enhanced the 
apparent performance of ASPs; not testing the limitations of ASPs’ detection 
capabilities-for example, by not using a sufficient amount of the type of mate- 
rials that would mask or hide dangerous sources and that ASPs would likely 
encounter at ports of entry; and not using a critical Customs and Border Pro- 
tection (CBP) standard operating procedure that is fundamental to the per- 
formance of hand-held radiation detectors in the field. 

• Recommendations. We recommended that the Secretary of Homeland Security 
delay Secretarial certification and full-scale production decisions on ASPs 
until all relevant tests and studies had been completed and limitations to 
tests and studies had been identified and addressed. We further rec- 
ommended that DHS determine the need for additional testing in cooperation 
with CBP and other stakeholders and, if additional testing was needed, that 
the Secretary of DHS appoint an independent group within DHS to conduct 
objective, comprehensive, and transparent testing that realistically dem- 
onstrates the capabilities and limitations of ASPs. 

Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy Advanced 
Radiation Detection Portal Monitors Is Likely to Exceed the Department’s Pre- 
vious Cost Estimates. GAO-08-1108R. Washington, D.C.: September 22, 2008. 

• Key findings. Our independent cost estimate suggested that from 2007 
through 2017 the total cost of DNDO’s 2006 project execution plan (the most 
recent official documentation of the program to equip U.S. ports of entry with 
radiation detection equipment) would likely be about $3.1 billion but could 
range from $2.6 billion to $3.8 billion. In contrast, we found that DNDO’s cost 
estimate of $2.1 billion was unreliable because it omitted major project costs, 
such as elements of the ASPs’ life cycle, and relied on a flawed methodology. 
DNDO officials told us that the agency was no longer following the 2006 
project execution plan and that the scope of the agency’s ASP deployment 
strategy had been reduced to only the standard cargo portal monitor. Our 
analysis of DNDO’s summary information outlining its scaled-back plan indi- 
cated the total cost to deploy standard cargo portals over the period 2008 
through 2017 would be about $2 billion but could range from $1.7 billion to 
$2.3 billion. Agency officials acknowledged the program requirements that 
would have been fulfilled by the discontinued ASPs remained valid, including 
screening rail cars and airport cargo, but the agency had no plans for how 
such screening would be accomplished. 

• Recommendations. We recommended that the Secretary of Homeland Security 
direct the Director of DNDO to work with CBP to update the projection exe- 
cution plan to guide the entire radiation detection program at U.S. ports of 
entry, revise the estimate of the program’s cost and ensure that the estimate 
considers all of the costs associated with its project execution plan, and com- 
municate the revised estimate to Congress so that it is fully apprised of the 
program’s scope and funding requirements. 

Combating Nuclear Smuggling: DHS Needs to Consider the Full Costs and Complete 
All Tests Prior to Making a Decision on Whether to Purchase Advanced Portal 
Monitors. GAO-08-1178T. Washington, D.C.: September 25, 2008. 

• Key findings. In preliminary observations of the 2008 round of ASP testing, 
we found that DNDO had made progress in addressing a number of problems 
we identified in previous rounds of ASP testing. However, the DHS criteria 
for significant increase in operational effectiveness appeared to set a low bar 
for improvement — for example, by requiring ASPs to perform at least as well 
as current-generation equipment when nuclear material is present in cargo 
but not specifying an actual improvement. In addition, the ASP certification 
schedule did not allow for completion of computer simulations that could pro- 
vide useful data on ASP capabilities prior to the Secretary’s decision on cer- 
tification. Finally, we questioned the replacement of current-generation equip- 
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ment with ASPs until DNDO demonstrates that any additional increase in se- 
curity would be worth the ASPs’ much higher cost. 1 

Combating Nuclear Smuggling: DHS’s Phase 3 Test Report on Advanced Portal 
Monitors Does Not Fully Disclose the Limitations of the Test Results. GAO-08- 
979. Washington, D.C.: September 30, 2008. 

• Key findings. DNDO’s report on the second group of ASP tests in 2007 (the 
Phase 3 tests) did not appropriately state test limitations. As a result, the re- 
port did not accurately depict the results and could potentially be misleading. 
The purpose of the Phase 3 tests was to conduct a limited number of test runs 
in order to identify areas in which the ASP software needed improvement. 
While aspects of the Phase 3 report addressed this purpose, the preponder- 
ance of the report went beyond the test’s original purpose and made compari- 
sons of the performance of the ASPs with one another or with currently de- 
ployed portal monitors. We found that it would not be appropriate to use the 
Phase 3 test report in determining whether the ASPs represent a significant 
improvement over currently deployed radiation equipment because the lim- 
ited number of test runs did not support many of the comparisons of ASP per- 
formance made in the report. 

• Recommendations. We recommended that the Secretary of DHS use the re- 
sults of the Phase 3 tests solely for the purposes for which they were in- 
tended — to identify areas needing improvement — and not as a justification for 
certifying whether the ASPs warrant full-scale production. If the Secretary in- 
tends to consider the results of the Phase 3 tests in making a certification 
decision regarding ASPs, we further recommended that the Secretary direct 
the Director of DNDO to revise and clarify the Phase 3 test report to more 
fully disclose and articulate the limitations present in the Phase 3 tests and 
clearly state which insights from the Phase 3 report are factored into any de- 
cision regarding the certification that ASPs demonstrate a significant increase 
in operational effectiveness. Finally, we recommended that the Secretary di- 
rect the Director of DNDO to take steps to ensure that any limitations associ- 
ated with the 2008 round of testing are properly disclosed when the results 
are reported. 

Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation De- 
tection Portal Monitors, but Preliminary Results Show Limits of the New Tech- 
nology. GAO-09-655. Washington, D.C.: May 21, 2009. 

• Key findings. We reported that the DHS criteria for a significant increase in 
operational effectiveness require a large reduction in innocent alarms but a 
marginal improvement in the detection of certain weapons — usable nuclear 
materials. In addition, the criteria do not take the current-generation portal 
monitors’ full potential into account because DNDO has not completed efforts 
to improve their performance. With regard to ASP testing, we found that 
DHS increased the rigor in comparison with previous tests, thereby adding 
credibility to the test results, but that preliminary results were mixed. The 
results showed that the new portal monitors performed better than current- 
generation portal monitors in detection of certain nuclear materials concealed 
by light shielding approximating the threat guidance for setting detection 
thresholds, but differences in sensitivity were less notable when shielding was 
slightly below or above that level. Testing also uncovered multiple problems 
in ASPs meeting the requirements for successful integration into operations 
at ports of entry. Finally, we found that DNDO did not plan to complete com- 
puter simulations that could provide additional insight into ASP capabilities 
and limitations prior to certification even though delays to testing allowed 
more time to conduct the simulations. 

• Recommendations. We recommended that the Secretary of Homeland Security 
direct the Director of DNDO to assess whether ASPs meet the criteria for a 
significant increase in operational effectiveness based on a valid comparison 
with current-generation portal monitors’ full performance potential and revise 
the schedule for ASP testing and certification to allow sufficient time for re- 
view and analysis of results from the final phases of testing and completion 
of all tests, including computer simulations. If ASPs are certified, we further 
recommended that the Secretary of Homeland Security direct the Director of 
DNDO to develop an initial deployment plan that allows CBP to uncover and 


iThis testimony also summarized our September 2008 report on the life cycle cost estimate 
to deploy ASPs (GAO-08-1108R). 
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resolve any additional problems not identified through testing before pro- 
ceeding to full-scale deployment. 

Biography for Gene Aloise 

Gene Aloise is a Director in the Natural Resources and Environment team at 
GAO. He is GAO’s recognized expert in international nuclear nonproliferation and 
safety issues and completed training on these subjects at the University of Virginia 
and Princeton University. His work for GAO has taken him to some of Russia’s 
closed nuclear cities and the Chernobyl reactor in Ukraine as well as numerous nu- 
clear facilities around the world and in the United States. Mr. Aloise has had years 
of experience developing, leading, and managing GAO domestic and international 
engagements. His diverse experience includes assignments with congressional com- 
mittees as well as various offices within GAO. He has received numerous awards 
for his leadership and expertise including GAO’s Meritorious Service Award. Mr. 
Aloise received his Bachelor’s degree in political science/economics from Rowan Uni- 
versity and holds a Master of Public Administration from Temple University. Mr. 
Aloise is also a graduate of the Senior Executive Fellows Program, John F. Kennedy 
School of Government, Harvard University. 

Chairman Miller. Thank you, Mr. Aloise. 

Dr. Lowenthal. 

STATEMENT OF DR. MICAH D. LOWENTHAL, DIRECTOR, NU- 
CLEAR SECURITY AND NUCLEAR FACILITY SAFETY PRO- 
GRAM, NUCLEAR AND RADIATION STUDIES BOARD, NA- 
TIONAL RESEARCH COUNCIL, THE NATIONAL ACADEMIES 

Dr. Lowenthal. Good morning, and thank you, Chairman Mil- 
ler, Ranking Member Broun, Members of the Committee. My name 
is Micah Lowenthal. As you said, I am the Director of the Nuclear 
Security and Nuclear Facility Safety Program at the National Re- 
search Council’s Nuclear and Radiation Studies Board. I am here 
to describe a recently issued interim report from a Congressionally 
mandated National Research Council study on Advanced 
Spectroscopic Portals. 

I am the study director supporting the authoring committee of 
that report, and I will begin by providing background on the re- 
quest for this study and then I will summarize the main messages 
of the report. 

The Department of Homeland Security, or DHS, wants to deploy 
new radiation detector systems called Advanced Spectroscopic Por- 
tals, or ASPs, to improve scrutiny of containerized cargo for nuclear 
and radiological material. The ASPs are intended to replace some 
or all of the radiation portal monitors (RPMs) and hand-held radio- 
isotope identifiers (RIIDs) currently used at ports and border cross- 
ings across the United States. Congress required that the Secretary 
of Homeland Security certify that the ASPs provide a significant 
increase in operational effectiveness over the old systems. Congress 
also directed DHS to ask the National Research Council to advise 
the Secretary on testing, analysis, costs and benefits of the ASPs 
before the certification decision. 

I want to point out that the appropriations committees who made 
these requirements said they appreciate that certification will be 
difficult. The study committee agrees with that, that these are hard 
problems, both for the testing and the cost-benefit analysis, so it 
is not surprising that DHS’s work has been challenging. 

Here are the main findings and recommendations from the in- 
terim report. The study committee found that the ASP performance 
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tests prior to 2008 had serious flaws. The Domestic Nuclear Detec- 
tion Office, DNDO, acknowledged problems with the tests and ad- 
dressed a number of those flaws in later testing. The 2008 perform- 
ance tests were indeed improved, but shortcomings remain. 
DNDO’s current approach to performance testing involves phys- 
ically testing detector performance against a small number of con- 
figurations of threat objects and cargo in one environment. The set 
of possible combinations of threats, cargo and environments is so 
large and multidimensional that DNDO needs an analytical basis 
for understanding the performance of its detector systems, not just 
an empirical basis. Our study committee recommended that DHS 
use a more rigorous approach in which scientists use computer 
models to simulate configurations and detector performance, use 
physical tests to validate and refine the models and use the models 
to select key new physical tests that advance our understanding of 
the detector systems. This iterative modeling and testing approach 
is standard in the development of some high-technology equipment 
and is essential for building scientific confidence in detector per- 
formance. 

The idea of an iterative approach also extends to deployment. 
The study committee recommends a process for incremental deploy- 
ment and continuous improvement of the ASPs with experience in 
the field leading to refinement in both the technologies and the op- 
erations for the next deployment. As a first step in this process, 
DHS should deploy the unused ASPs it already has, to assess the 
ASPs’ performance in multiple environments without investing in 
a much larger acquisition at the outset. 

The ASP cost-benefit analysis was not complete when our in- 
terim report was written. Preliminary estimates by DNDO indicate 
that the cost increases from replacing the currently deployed sys- 
tems with ASPs outweigh the cost reduction or savings from oper- 
ational efficiencies. Therefore, a careful cost-benefit analysis will 
need to reveal the security advantages, if any, of the ASPs over the 
current systems and possible alternatives. Such a cost-benefit anal- 
ysis should include three key elements: a clear statement of the ob- 
jectives of the program, an assessment of meaningful alternatives; 
and a comprehensive, credible and transparent analysis of benefits 
and costs. 

DHS should consider tradeoffs among different options for allo- 
cating its efforts and funds, looking at the overall system for ways 
to improve defense against nuclear smuggling. The study com- 
mittee recommended that DHS not proceed with further procure- 
ment of ASPs until it has addressed the findings and recommenda- 
tions from the report, and the ASP has been shown to be a favored 
option in the cost-benefit analysis. 

Now, those are the main messages from the report. For this 
hearing I was also asked to comment on any lessons learned re- 
garding processes by which the ASPs have been researched, devel- 
oped and tested to date. Although the study committee only exam- 
ined the testing of ASPs, not a broader portfolio, I think there are 
three lessons to be learned based on the study. First, the process 
of modeling and testing iteratively can be applied more broadly to 
other complex technology development programs. Second, incre- 
mental deployment with continuous improvement is a good strat- 
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egy for deployments of systems that have not fully matured, espe- 
cially if they are envisioned to have an ongoing mission. And third, 
the systems-level approach that is examining how to optimize 
choices within the overall system, rather than focusing narrowly on 
one tradeoff decision, is applicable to almost every use of equip- 
ment in security applications. 

Thank you for the opportunity to testify. I would be glad to 
elaborate my testimony in response to questions. 

[The prepared statement of Dr. Lowenthal follows:] 

Prepared Statement of Micah D. Lowenthal 

Good morning Chairman Miller, Ranking Member Broun, and Members of the 
Committee. My name is Micah Lowenthal and I am the Director of the program on 
Nuclear Security and Nuclear Facility Safety in the National Research Council’s 
Nuclear and Radiation Studies Board. 1 I am here to describe the recently issued in- 
terim report from a congressionally mandated National Research Council study on 
advanced spectroscopic portals (ASPs). I am the study director supporting the au- 
thoring committee of that report. 2 The full report is classified, but an abbreviated 
version was also produced for unrestricted public release. 3 My testimony is based 
on the abbreviated version. I will begin by providing background on the request for 
this study. I will then summarize the main messages of the report and discuss some 
of the points most relevant to this hearing. 

BACKGROUND ON THE REQUEST FOR THE STUDY 

Containerized cargo entering the United States at sea ports and land-border 
crossings for trucks is currently screened for radiation using detectors, called radi- 
ation portal monitors (RPMs), in conjunction with hand-held radioisotope identifiers 
(RIIDs). The Department of Homeland Security (DHS) is seeking to deploy new ra- 
diation detectors, called advanced spectroscopic portals (ASPs), to replace the cur- 
rent RPM and RIID combination, which has known deficiencies. The ASPs consist 
of new detector equipment and new software, including algorithms for isotope iden- 
tification. 

Following some controversy over the testing and evaluation of the new ASPs, Con- 
gress required in Title IV of Division E of the Consolidated Appropriations Act, 2008 
(Public Law 110-161) that the Secretary of Homeland Security submit to Congress 
a report certifying that ASPs would provide a “significant increase in operational 
effectiveness” over continued use of existing screening devices. This certification is 
a precondition for proceeding with full-scale procurement of ASPs. Congress also di- 
rected DHS to request that the National Academies advise the Secretary on the cer- 
tification decision by helping to validate testing completed to date, providing support 
for future testing, assessing the costs and benefits of this technology, and bringing 
robustness and scientific rigor to the procurement process. Due to delays in the test 
and evaluation program, the Academies and DHS agreed that the study committee 
would issue an interim report that provides ( 1 ) the committee’s evaluation of testing 
plans and execution it has seen, and (2) advice on how the Domestic Nuclear Detec- 
tion Office (DNDO) can complete and make more rigorous its ASP evaluation for 
the Secretary and the Nation. 

This interim report is based on testing done before 2008 (referred to as past tests), 
plans for and preliminary results from performance tests carried out in 2008, and 
the agency’s draft cost-benefit analysis as of October 2008. The committee received 
briefings on the performance test results and analysis and on the cost-benefit anal- 
ysis but it did not receive any written reports on those topics by February 2009, 
when the interim report entered the Academies peer review process. 

I will now discuss each element of the study task below. 


1 The National Research Council is the operating arm of the National Academy of Sciences, 
the National Academy of Engineering, and the Institute of Medicine of the National Academies, 
chartered by Congress in 1863 to advise the government on matters of science and technology. 
The Nuclear and Radiation Studies Board is responsible for oversight of National Research 
Council studies on safety and security of nuclear materials and waste. 

2 Dr. Robert Dynes, a physicist at the University of California, member of the National Acad- 
emy of Sciences, and former President of the University of California, chaired this study. 

3 The report is titled Evaluating Testing, Costs, and Benefits of Advanced Spectroscopic Portals 
for Screening Cargo at Ports of Entry: Interim Report. The abbreviated version of the report is 
available online at http: // www.nap.edu / catalog.php?record -id=XXXX 
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SUMMARY OF THE MAIN MESSAGES OF THE REPORT 

First, I want to note that the committee focused much of its attention on perform- 
ance testing. This is not because the other tests are unimportant — the portals will 
be of little use if they are incompatible with CBP’s computer systems, for example — 
but the design, execution, and evaluation of these tests are comparatively routine, 
even if solutions to problems revealed by the tests are not. The design, execution, 
and evaluation of performance tests for the ASPs is more challenging and involves 
more of the science and engineering principles on which the committee has advice 
to offer. 

Past Performance Testing 

Performance tests prior to 2008 had serious flaws that were identified by the Gov- 
ernment Accountability Office and the Secretary’s ASP Independent Review Team. 
All truck-conveyed containers at ports and border crossings currently pass through 
primary screening, which is conducted with a radiation portal monitor (RPM). Con- 
tainers that trigger an alarm are sent to secondary screening, which is conducted 
with an RPM and a RIID. The tests prior to 2008 did not adequately assess the ca- 
pabilities of the ASP systems in primary and secondary screening compared with 
the currently deployed RPM and RIID screening systems, nor whether the ASP sys- 
tems met performance criteria for procurement. 

There were serious flaws in the testing protocol. Notably, DNDO utilized the same 
radiation sources in performance testing that were used to set up and calibrate this 
testing. Device setup and any calibration must use separate radiation sources from 
those used for testing. Also, standard operating procedures for the use of RlIDs in 
secondary screening were not followed in the performance tests, which disadvan- 
taged the RIIDs in comparisons with ASPs. 

2008 Performance Testing 

DNDO staff acknowledged several pre-2008 deficiencies and designed its 2008 test 
plan to correct them. The study committee examined the revised test plan, observed 
tests, and questioned test personnel, and the committee concluded that DNDO did 
address those problems. 

Because of the ASP configurations and the size of their detectors, ASPs would be 
expected to improve isotope identification, provide greater consistency and coverage 
in screening, , and increase speed of screening compared to the current RPM-RIID 
combination when used in secondary screening. Consequently, DNDO’s 2008 per- 
formance tests of ASPs in secondary screening focused on confirming and quanti- 
fying that advantage for several threat objects, cargoes, and configurations. 

When used for primary screening, an ASP system must be compared to the exist- 
ing RPM-RIID combination for primary and secondary screening. This is because 
ASPs perform an isotope identification function in primary screening. Isotope identi- 
fication is only possible in secondary screening with the current RPM-RIID system. 
DNDO’s preliminary analysis did account for this difference. 

The study committee found that the 2008 performance tests were an improvement 
over previous tests. They enabled DNDO to identify and physically test some of the 
performance limits of ASP systems. However, the committee identified several 
shortcomings of these tests: (1) The selected test configurations were too limited to 
assess the performance of ASP systems against the range of threat objects, cargoes, 
and configurations that could be encountered during cargo screening operations at 
ports without modeling to complement the physical experiments; (2) the sample 
sizes (the number of test runs of each case) are small and limit the confidence that 
can be placed in comparing ASP and RPM-RIID performance; and (3) in its anal- 
ysis, some of the performance metrics are not the correct ones for comparing oper- 
ational performance of cargo screening systems. These shortcomings are described 
in greater detail in our report. In the committee’s judgment, DHS cannot determine 
whether ASPs can consistently outperform current RPM-RIID systems in routine 
practice until these shortcomings are addressed. Better physical measurement and 
characterization of the performance of the systems are a necessary first step but 
may not be sufficient to enable DHS to conclude that the ASPs meet the criteria 
it has defined for achieving a “significant increase in operational effectiveness.” 

The committee recommends modifications to the testing procedures that are being 
used by DHS. These modifications would influence subsequent procurement steps, 
as described in the recommendations for the procurement process. 

Recommended Approach for Testing and Evaluation 

To make the testing and evaluation more scientifically rigorous, the committee 
recommends an iterative approach involving modeling and physical testing. The 
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threat space that is, the set of possible threat objects, configurations, surrounding 
cargoes, and conditions of transport — is so large and multi-dimensional that DNDO 
needs an analytical basis for understanding the capabilities of ASP detectors for 
screening cargo. DNDO’s current approach is to physically test small portions of the 
threat space and to use other experimental data to interpolate and extrapolate to 
other important parts of the threat space to test the identification algorithms in 
ASP systems. 

The committee recommends that DNDO use computer models of threat objects, 
radiation transport, and detector response to simulate ASP performance. Then 
DNDO can use physical experiments to validate these computer models, which 
would allow a critique of the models’ fidelity to reality and show where model refine- 
ments were needed. Physical testing and model refinement would proceed iteratively 
until the model provided an acceptably accurate depiction of reality. With validated 
models, DNDO could evaluate the performance of ASP systems over a larger, more 
meaningful range of the threat space than is feasible with physical tests alone. 

This kind of interaction between computer models and physical tests is standard 
in the development and deployment of some high-technology equipment and is es- 
sential for building scientific confidence in technology performance. The performance 
tests conducted in 2008, and even prior to 2008, can be used to help refine and vali- 
date models. The committee also notes the skills required to proceed exist in the 
National Laboratories. 

Recommended Approach for the Procurement Process 

The idea of an iterative approach also extends to deployment of ASP systems at 
ports of entry. The committee noted that DHS’ testing philosophy is oriented toward 
a one-time certification decision in the near future. However, the mandate for pas- 
sive radiation screening of cargo at ports of entry is expected to continue indefi- 
nitely. Rather than focusing on a one-time decision about the deployment of ASPs, 
the committee suggests that current testing be viewed as a first step in a continuous 
process of system improvement and adaptation to changes in the threat environ- 
ment, composition of container cargo, technological and analytical capabilities, and 
the nature of commerce at ports of entry. These factors have changed significantly 
over the last decade and can be expected to evolve — in both predictable and unpre- 
dictable ways — in the future. The committee recommends that DHS should develop 
a process for incremental deployment and continuous improvement, with experience 
leading to refinements in both technologies and operations over time, rather than 
a single product purchase to replace current screening technologies. The ASP de- 
ployment process should be developed to address and exploit changes. This would 
enable DNDO to adapt and continually update its screening systems so that they 
do not become outdated as they would after a one-time deployment. 

As the first step in this process, the committee recommends that DHS deploy its 
currently unused low-rate initial production ASPs for primary and secondary inspec- 
tion at various sites under a program of extended operational testing. Such deploy- 
ment, even on a limited scale, would provide valuable data concerning ASP oper- 
ation, reliability, and performance, and would allow DHS to better assess ASP capa- 
bilities in multiple environments without investing in a much larger acquisition at 
the outset. 

The development of the hardware for radiation detection and the software for ana- 
lyzing detector signals is separable. The current DHS procurement process is a com- 
petition among vendors to provide the combined systems, which has been useful. 
However, as DHS moves forward, the committee recommends that it match the best 
hardware to the best software (particularly the algorithms), drawing on tools devel- 
oped by the competing companies and others, such as the national laboratories. 

The deployment of ASPs will not eliminate the need for hand-held detectors with 
spectroscopic capabilities. Because some of the improvement in isotope identification 
offered by the ASPs over the RIIDs is a result of software improvements, the com- 
mittee recommends that these improvements be incorporated into hand-held detec- 
tors. Improved software might significantly improve RIID performance and expand 
the range of deployment options available to Customs and Border Protection for 
cargo screening. 

By separating the hardware and software elements of the system and engaging 
the broader science and engineering community, DHS would have increased con- 
fidence in its procurement of the best product available with current technology, and 
simultaneously could advance the state-of-the-art. 
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Recommended Approach for Cost-Benefit Analysis 

The preliminary analysis presented to the committee suggests that benefits of de- 
ploying the ASPs may not be clearly greater than the costs. Because DNDO’s pre- 
liminary estimates indicate that the cost increases from replacing the RPM-RIID 
combination with ASPs exceed the cost reductions from operational efficiencies, it 
is important to consider carefully the conditions under which the benefits of deploy- 
ing ASPs justify the program costs. A cost-benefit analysis (CBA) can provide a 
structure for evaluating whether a proposed program (such as the ASP program) is 
reasonable and justified. 

The Secretary’s decision on ASP certification is based, at least in part, on whether 
the ASPs meet the objectives in DHS’ definition of “significant increase in oper- 
ational effectiveness” (SlOE); however, other factors relating to the costs and bene- 
fits of the proposed ASP program will also need to be taken into account. DHS’ defi- 
nition of SlOE is a modest set of goals: As noted above, the increases in operational 
efficiency do not by themselves appear to outweigh the cost increases from replacing 
the RPM/RIID combination with ASPs, based on DNDO’s preliminary estimates, 
and the criteria do not require a significantly improved ability to detect special nu- 
clear material (an ingredient of a nuclear weapon) in primary screening. If the ASPs 
meet the defined goals and are able to detect the minimum quantities of nuclear 
threat material recommended by the “DOE guidance,” DHS still will not know 
whether the benefits of the ASPs outweigh the additional costs associated with 
them, or whether the funds are more effectively spent on other elements of the 
Global Architecture. 

A CBA can provide insight about alternative choices — for example, whether the 
benefits of a given program exceed its costs, and which choices are most cost-effec- 
tive. To be effective, the CBA must include three key elements: (1) a clear statement 
of the objectives of the screening program; (2) an assessment of meaningful alter- 
natives to deploying ASPs; and (3) a comprehensive, credible and transparent anal- 
ysis of in-scope benefits and costs. 

The CBA should begin with a clear statement of what operational problem the 
ASPs are intended to address. This statement will define the role that the system 
plays in providing a layer in the defense against the importation of nuclear or radio- 
logical materials. The CBA should also include a narrative that clarifies how im- 
proving detection for containers at ports of entry to the United States fits into a 
larger effort to improve detection capabilities, in recognition of the many ways that 
materials could be brought into the United States through ports of entry that are 
not already screened, or across uncontrolled stretches of border. Furthermore, to be 
useful in a procurement decision, a CBA must address whether funds are better 
spent to replace the currently deployed equipment rather than to expand coverage 
for other material pathways that currently have no radiation screening. DHS should 
consider tradeoffs among different options for allocating efforts and funds, looking 
at the overall system for ways to improve defenses against nuclear smuggling. Such 
an analysis is needed in the CBA for ASP systems because it is not evident that 
it has been provided elsewhere. 

The CBA also needs to account for meaningful alternatives (including non-ASP 
systems) to reveal the scale of the benefits of ASPs for radiation screening and de- 
termine whether these benefits outweigh the additional costs for procurement and 
deployment. The complexity of the container screening task suggests that there 
could be many different options worthy of consideration. These options include vari- 
ations on ASP deployment configurations and operational processes, and application 
of technologies beyond the current RPM-RIID and ASP systems such as deploying 
hand-held passive detectors with state-of-the-art software and advanced methods for 
detecting nuclear materials. Considerations should include active interrogation, im- 
proved imaging systems, and integration of these existing technologies. These alter- 
natives need to be compared to a baseline that reflects as realistically as possible 
the screening capability that DHS currently has in place. This baseline should re- 
flect the number and placement of current RPM and RIID detectors, sensitivity of 
these detectors based on how they are operated at each port, and performance of 
existing hand-held detectors in the manner they are used in the field. The CBA 
must indicate what capability an investment in ASPs will provide beyond the exist- 
ing systems as they are currently deployed and operated, or beyond alternative radi- 
ation detection technologies that could be developed and deployed at ports of entry. 

In comparing alternatives, it is important that the CBA treat benefits and costs 
in a comprehensive, credible, and transparent manner. The benefit assessment 
should show how the ASP system would contribute to improving security with re- 
spect to prevention of the detonation of a nuclear device or radiological weapon in 
the United States. Because this is the primary objective of the ASP program, a CBA 
that is silent on this subject would be incomplete. Such an assessment is difficult 
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and no assessment of such benefits will be definitive or unassailable. The cost as- 
sessment should cover all phases of the acquisition life cycle in a manner that is 
independent of contractor or program office biases, and it should also assess the risk 
of cost escalation associated with the estimate. 

The committee recommends that DHS not proceed with further procurement until 
it has addressed the findings and recommendations in this report, and then only if 
the ASP is shown to be a favored option in the CBA. 

“Lessons Learned” 

For this hearing I was also asked to comment on “lessons learned” regarding the 
processes by which the ASPs have been researched, developed, and tested. Although 
the study committee only examined the testing of ASPs, my personal view is that 
three lessons that could be learned: 

First, the process of iterative modeling and testing can be applied more broadly 
to other complex technology development programs. Modeling coupled to validating 
experiments is a necessity for some technology applications because of the complex 
conditions in which these technologies must operate. 

Second, incremental deployment with continuous improvement is a good strategy 
for deploying systems that have not fully matured, especially if they are envisioned 
to have an ongoing mission. 

Third, a systems-level approach — that is, examining how to optimize choices with- 
in the overall system rather than narrowly focusing on one tradeoff decision — is ap- 
plicable to almost every use of equipment in security applications. 

This concludes my testimony to the Committee. Thank you for the opportunity to 
testify on this important topic. I would be happy to elaborate on any of my com- 
ments during the question and answer period. 

Biography for Micah D. Lowenthal 

Micah Lowenthal is the Director of the Nuclear Security and Nuclear Facility 
Safety Program in the Nuclear and Radiation Studies Board at the National Re- 
search Council of the National Academies. At the Academies, he has directed domes- 
tic and international studies on nuclear and radiological safety and security, nuclear 
nonproliferation, nuclear fuel cycles, radioactive waste, and management of contami- 
nated environments. Before joining the National Academies’ staff in 2001, Dr. 
Lowenthal was a researcher and lecturer at the University of California at Berkeley. 
In 1996 he was an American Association for the Advancement of Science Environ- 
mental Science and Engineering Fellow. Dr. Lowenthal received an A.B. degree in 
physics and a Ph.D. degree in nuclear engineering, both from the U.C.-Berkeley. 

Chairman Miller. Thank you, Dr. Lowenthal. Unfortunately, 
you all probably know that our days here are somewhat scattered, 
and I need to go to the Floor. We have tried to arrange another 
Member of the Majority who is a Member of the Subcommittee to 
preside in my absence. Ms. Dahlkemper has been an especially con- 
scientious Member and she may arrive between now and 11:00, 
which is when I expect to come back, and if that is the case, we 
will come out of recess and she will preside or any other Member 
of the Majority who is a Member of the Subcommittee. 

Mr. Broun. Mr. Chairman, I would be glad to preside if you 
would like me to. 

Chairman Miller. Well, I thank Dr. Broun for that generous 
offer but I think we will do it the way that every other committee 
and subcommittee in Congress does it. So with that, we will need 
to stand in recess. I apologize again for the inconvenience. Believe 
me, it is more inconvenient for me than it is for you, and we will 
be back in session but I appreciate your being here and I thank you 
for your indulgence. 

[Recess.] 
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Discussion 

Chairman Miller. We are now out of recess. Floor colloquies are 
undoubtedly one of the most peculiar stylized aspects of Congres- 
sional service. I have heard of political campaigns that 80 percent 
of all the effort is wasted, the problem is, no one knows what the 
20 percent is that is not. I suspect that there is at least that ratio 
of wasted effort in Congressional service and I very much suspect 
Floor colloquies are in the wasted effort category. But it was some- 
thing I had to do. 

Mr. Broun. Floor debate, isn’t it? 

Chairman Miller. I will resist temptation to point out other as- 
pects of Congressional service that may be wasted efforts. If we 
may now begin our rounds of questioning, Mr. Aloise, Dr. 
Lowenthal, Dr. Broun, I now recognize myself for five minutes. 

Prioritizing Security Needs 

Dr. Broun raised the point that there are other priorities that we 
have to decide between, priorities even within security, so it is not 
security versus other things government does. In October of 2007, 
this committee held a hearing on an exercise on a scenario of a ter- 
rorist attack assuming that a dirty bomb was detonated in two 
American cities, and what we found was that we were probably 
better prepared, certainly better prepared for Katrina than we 
were for dirty-bomb attacks, despite the fact that if there is one 
punch that has obviously been telegraphed by terrorists, by al- 
Qaeda in particular, it is a dirty-bomb attack. We did not have the 
capacity to conduct the test to see what the level of contamination 
was, to let our people know that they can go back into offices, office 
buildings — that we might shut down, indefinitely, 10, 15, 20 square 
blocks of an American city, the midtown of an American city or per- 
haps two — that would probably take six years to complete the nec- 
essary environmental tests for an attack in one city, nor testing on 
individuals to see if they had been exposed to radiation. That might 
take a couple years. We would have to tell parents that we could 
not test their children to see if they had been exposed to radiation 
for a couple years. But, you know, if the child’s hair started falling 
out, call us and maybe we can move them up in line. And the fail- 
ing there seemed to be simply one of funding. We are not prepared 
to respond to a radiological attack, and here we see we are spend- 
ing $2 or $3 billion to develop a technology that those who will use 
it really question whether it does much if any good at all, certainly 
not enough to justify $2 or $3 billion. DNDO expects the Secretary 
of Homeland Security to make a decision this October, four months 
from now, about whether to certify that it is worth investing the 
$2 billion in the ASPs. Do either of you believe that there will be 
enough information available this fall for DHS to make that deci- 
sion, and how confident are you that the costs of the ASPs justify 
the investment? Mr. Aloise. 

Mr. Aloise. Well, that is the question. In our view, until integra- 
tion testing is done, field validation is done, operational testing is 
done, the injection studies are complete, an analysis of energy 
windowing versus the ASPs is done, until all the testing is done 
and we know whether or not this marginal increase in security is 
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justified through a sound, independent cost-benefit analysis, it is 
our belief that we shouldn’t spend any money on this program or 
go forward with it. 

Chairman Miller. And do you think there is any realistic chance 
those preconditions may be satisfied between now and October? 

Mr. Aloise. At this point it does not look that way. 

Chairman Miller. Dr. Lowenthal. 

Dr. Lowenthal. We agree that these are the preconditions for 
going ahead with certification, that there is a great deal of informa- 
tion that is needed. I am not really in a position to tell you how 
much time it will take them to get that information together. There 
is some information that they presumably have already put to- 
gether, some analysis that they have already done that we haven’t 
seen. We will be happy to look at it when they provide it. But in 
terms of timelines, we would not be inclined to guess when they 
will be ready. 

Decision-making: Processes and Timelines 

Chairman Miller. Both of you appear to disagree with DHS’s 
view that they will be able to obtain the data they need from field 
evaluation tests that are to begin next month. Why do you not be- 
lieve that those tests will be good enough for DHS to base a deci- 
sion on? Mr. Aloise. 

Mr. Aloise. Well, that is only part of, as I just said, what they 
need to base their decision on. You know, field validation testing 
has not gone well so far. They suspended it after two weeks be- 
cause they were actually sending more — the ASPs were actually 
sending more to secondary than the PVTs so they had to work out 
a number of those problems and I am assuming they have. It has 
been a while since the initial testing. We just have to see how it 
goes, and we are going to be looking to CBP for a lot of the answers 
because they are the agency that has to live with this equipment, 
and I — we feel fairly confident that they are not going to go for- 
ward until they are satisfied that the testing has gone right at this 
point. 

Chairman Miller. Dr. Lowenthal. 

Dr. Lowenthal. The testing that they are going to do for field 
validation is going to tell them something that they need to know 
but it won’t tell them everything that they need to know, and the 
committee — the study committee’s recommendations were to en- 
hance what they are going to be able to learn from these sorts of 
tests with modeling and simulations. The range of environments 
that they are going to be operating in is very limited. They are only 
doing field validation testing in four different sites, and there are 
a lot of other conditions, a lot of other kinds of cargo and so on that 
are worth learning about. Some of that can be done with simula- 
tion and some of it can be done with the deployment that the com- 
mittee recommended. 

Chairman Miller. Thank you. My time has expired. 

Dr. Broun for five minutes. 

Mr. Broun. Thank you, Chairman. 
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ASP in Context: The Global Nuclear Detection 
Architecture 

How would y’all — “y’all” is a southern word. How would you all 
prioritize the upgrade from the PVT RIID system to the ASP sys- 
tem within the entire global nuclear detection architecture? Let us 
start with Dr. Lowenthal. 

Dr. Lowenthal. I should point out that the committee really 
only looked at the testing of the ASPs and talked about what kinds 
of information a decision-maker would want to have; and so the 
committee recommended that they (DHS) consider these other com- 
ponents of the global architecture — and there are gaps that are 
known and have been pointed out by others within that architec- 
ture — but the committee did not study that question, itself. What 
the committee did say is that if you look at this from the perspec- 
tive of an adversary who is trying to sneak something into the 
United States, if you can — if you reinforce the places where you al- 
ready are screening but you leave other avenues totally open, you 
may not be gaining much improvement in security. 

Mr. Broun. Mr. Aloise. 

Mr. Aloise. We agree. As I mentioned to Dr. Lowenthal a 
minute ago, we are spending a lot of money upgrading locks on the 
front door but the windows and the back door are wide open. Does 
that make sense to do with limited resources, so 

Mr. Broun. I agree wholeheartedly. Are there other areas that 
would benefit greater from the same level of investment, Mr. 
Aloise? 

Mr. Aloise. Well, there are a lot of vulnerabilities that need to 
be looked at that have not been properly addressed in the architec- 
ture. The border crossing is really not the greatest threat, so I 
would — we believe that we should be looking equally hard at all 
the other vulnerabilities in the architecture because as it was just 
said, that is probably not where they are going to come through. 
If they have taken the time and trouble to get this material, it is 
unlikely they are going to stick it in a cargo container and drive 
it across through a portal monitor. 

Mr. Broun. And with open borders, they can just carry it in 
wherever they want to carry it. 

Dr. Lowenthal, particular — and let me ask you to answer your 
question in view of comments you made in your testimony about 
that we have ASPs sitting there now, if you would include that dis- 
cussion? 

Dr. Lowenthal. Well, given the scope of our study, I wouldn’t 
be comfortable to try to identify what the most promising area is. 
What the committee recommended is that DHS do that analysis, 
look at the entire system. We have some concerns that that hasn’t 
been done, and — I will separate this into two parts. One is that the 
committee recommended that DHS look at that whole system in 
addition to just the tradeoff between the ASPs and the PVTs, be- 
cause as a decision-maker, the people in the Administration and in 
Congress will want to know whether the money is well spent. The 
analysis does not appear to have been done elsewhere and so the 
committee recommends that it be included in this ASP decision. 
Now, the other component of that is the deployment of the ones 
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that they already own. That is more for a learning exercise. This 
is something where they can learn more about the performance of 
the systems in the field and improve them as they go forward. I 
think these are directed at slightly different points. 

Mr. Broun. Thank you. My time disappeared, so I guess I am 
out of time. I am not sure where we stand. Thank you, Mr. Chair- 
man. 

Chairman Miller. Well, you will have time for a second round 
because I was planning to ask at least one more question. I now 
recognize myself for a second round of questions. 

Management of ASP Testing and Deployment 

Your criticism, for both of you, the criticism appears to be as 
much that the problems with the ASPs was not the technical chal- 
lenges but that the whole program was just poorly managed. This 
is a recurring theme for this committee, you know, how could man- 
agement — how could programs be managed better. But if you think 
the whole testing program has been troubled from the start, and 
that the management more than anything else is a problem, what 
lessons should we learn? How should this program be managed dif- 
ferently? How can it be managed differently in the future, this and 
other similar programs? 

Mr. Aloise. Well, I will take a shot at that. Back in 2006 when 
we issued our first report on the deployment of PVTs, we learned 
about the ASPs and we knew then that there would only be a mar- 
ginal improvement in security. That is why we called for a cost- 
benefit analysis before anything went further; is it worth the mar- 
ginal increase in security you are going to get from the ASPs? Do 
that first, then go talk to the Congress and the decision-makers can 
decide whether it is worth it or not, and we know the story. It was 
poorly done, and it was based on assumptions and not data, and 
that leads us to where we are today. It was a rush to push tech- 
nology through that was immature, and that has got to be the true 
lesson of this, is that without proper and scientifically rigorous 
testing, without a technology that is mature, it should not be gen- 
erally rushed to deployment — especially when you have a system 
already there that is working. 

Chairman Miller. Right. Dr. Lowenthal. 

Dr. Lowenthal. I should point out that our study committee was 
silent on the matter of whether this was managed well or not, but 
I think that the report highlights many of the points that you made 
in your opening remarks about what should be done in the future. 
I think you hit on all of those — a well-planned-out testing program 
with criteria established in advance and an idea as you go in for 
what you are going to learn from each stage of this, making sure 
that it matches up with what the customer is looking for, as Mr. 
Aloise mentioned. All of those components lead to having a testing 
and evaluation program that will lead to procurement that gives 
the customer what he needs and the Nation what it needs. 

Chairman Miller. Would it be a fair summary to say that the 
problem is that we began with the conclusion and then found the 
analysis to support it, rather than beginning with the analysis, and 
then finding what conclusions would come from the analysis? 
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Mr. Aloise. I think so, Mr. Chairman. I think we had a solution 
in search of a problem. 

Chairman Miller. There is a problem but there may have been 
another solution. Our clock is not working, so each Member needs 
to show self-restraint. I will now go to Dr. Broun for a second 
round. Ms. Dahlkemper has arrived and I think is now collecting 
herself. 

Dr. Broun for five minutes, however that might be measured. 
Five minutes based on the honor system. 

Mr. Broun. I hope Ms. Dahlkemper can collect herself. I trust 
that she can. She is very good at doing that. 

I am very interested to hear Dr. Lowenthal’s comments about 
whether we can take the ASP units that are available now, put 
them in a parallel screening process out there in the field, as the 
appropriators evidently are preventing us from doing. Would you 
recommend that we do so, and is that something that you could put 
in your report to us, so that we can look at that possibility and 
maybe even make that recommendation? 

Dr. Lowenthal. Yes. Our study committee recommended that 
they take the — they procured a number of these low rate initial 
production [units] so that they can test and evaluate them. They 
have a number of them that have been sitting moth-balled. I un- 
derstand that some of them have been cannibalized for parts for 
other ones, and we don’t have an exact number of how many they 
have, but they have some number of them that they already own 
that they could deploy in the field. Our study committee’s rec- 
ommendation, explicitly in the report, is that these should be de- 
ployed at selected sites, and the experience gained from those will 
contribute to both their understanding of how they perform, but 
also how they might be able to modify the operations at the sites, 
and also the technology, what improvements they might want to 
put in there. So that is explicitly in the report. 

Mr. Broun. Mr. Aloise, any comments? 

Mr. Aloise. Just that we would agree with that. We think that 
is a smart idea. 

Mr. Broun. Thank you. I hope that the CBP people can be con- 
sulted because I think the end-user has not been consulted enough 
in this process, and with that, Mr. Chairman, I will yield back. 

Chairman Miller. Thank you. You may or may not have had 
time to yield back. You probably did. 

Ms. Dahlkemper for more or less five minutes. 

Ms. Dahlkemper. Thank you very much, and I apologize for 
being late. I had a markup in another committee, but I appreciate 
the opportunity to ask a question of both of you if you could just 
address this. 

Justifying the Cost of the Advanced Spectroscopic 
Portal Program 

Can either of you provide some insight into whether improving 
portal radiation detection is worth $2 billion to $3 billion? Assum- 
ing that ASPs prove to be robust enough to use, and they deliver 
real improvement in detection in commercial flow, would that still 
be enough to justify deploying them, given all the other needs out 
there and our limited resources? 
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Mr. Aloise. I have to go back to, and I know I sound like a bro- 
ken record on this, a cost-benefit analysis that we have been calling 
for a couple years, although those factors that you mentioned need 
to be in there so that the decision-makers can look at and see 
whether the marginal improvement is worth it, considering all the 
vulnerabilities we have in the architecture already that we need to 
address. 

Ms. Dahlkemper. Dr. Lowenthal. 

Dr. Lowenthal. We agree entirely that a cost-benefit analysis 
will be needed to show that. We are not in a position to judge 
whether they are going to be worth it, partly because our study 
didn’t examine the results of their tests for this interim report. It 
looked at the approach. Once they have the results and they can 
factor those into their cost-benefit analysis, then a judgment can be 
made as to whether they are worth it. 

Ms. Dahlkemper. Mr. Aloise, the GAO has done a lot of work 
on how agencies should do a cost-benefit analysis. Can you lay out 
some principles that should be guiding DHS as they begin their 
own cost-benefit analysis on whether to acquire ASPs or not? 

Mr. Aloise. Yes. In fact, we just developed this about a year ago 
on how to develop a cost-benefit analysis. It was based on working 
with 90 representatives from all the federal agencies including 
DHS and it lays out exactly what the best practices are. But some 
of the things must be based on fact, must be fact based, based on 
data, not assumptions, must have proper documentation all along 
the way, and those are the kinds of things that we are missing 
from the ones we looked at for DNDO. 

Ms. Dahlkemper. Anything specific to this particular 

Mr. Aloise. In particular, test results, what do the test results 
show. 

Ms. Dahlkemper. All right. Dr. Lowenthal, do you want to com- 
ment on that? 

Dr. Lowenthal. If I can just add one point, we give some advice 
on how to complete the cost-benefit analysis but one of the things 
that is pointed out in our report is that no cost-benefit analysis at 
the end that deals with security matters is going to be totally unas- 
sailable. They are going to have to produce something that they 
can defend, but you are always going to be able to pick at some 
part of it. This is a difficult problem. This is not an easy issue to 
work through. 

Ms. Dahlkemper. Thank you. I yield back. 

Chairman Miller. Thank you, Ms. Dahlkemper. You still had 
five minutes. 

That concludes the questioning of this first panel. Again, we 
thank both Mr. Aloise and Dr. Lowenthal, and we will stand at 
ease for a second while this panel can step down and the next 
panel can come up. 


Panel II: 

At this time I would like to introduce our second panel. Dr. Wil- 
liam Hagan is the Acting Deputy Director of the Domestic Nuclear 
Detection Office at the Department of Homeland Security. That is 
also a mouthful of a title. Mr. Todd C. Owen is the Acting Deputy 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00040 Fmt6633 Sfmt6601 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



35 


Assistant Commissioner of the Office of Field Operations at the 
U.S. Customs and Border Protection, Department of Homeland Se- 
curity. You each have five minutes for your spoken testimony. Your 
written testimony will be included in the record. When we have 
completed your spoken testimony, we will begin with questions. 
Each Member will have five minutes to question the panel. It is the 
practice of this subcommittee to receive testimony under oath. Do 
either of you have any objection to taking an oath? Both witnesses 
indicated that they do not. If not, you also have the right to be rep- 
resented by counsel. Do either of you have counsel here? And both 
indicated they do not have counsel present. Please stand and raise 
your right hand. Do you swear to tell the truth and nothing but 
the truth? 

The record should reflect that both witnesses took the oath. We 
will begin with Dr. Hagan. 

STATEMENT OF DR. WILLIAM K. HAGAN, ACTING DEPUTY DI- 
RECTOR, DOMESTIC NUCLEAR DETECTION OFFICE, DE- 
PARTMENT OF HOMELAND SECURITY 

Dr. Hagan. Good morning, Chairman Miller, Ranking Member 
Broun and distinguished Members of the Subcommittee. As Acting 
Deputy Director of the Domestic Nuclear Detection Office, DNDO, 
at the Department of Homeland Security, I am honored to be here 
with my colleague from U.S. Customs and Border Protection (CBP), 
Mr. Todd Owen. I would like to thank the Committee for the oppor- 
tunity to share lessons learned and progress to date on our Ad- 
vanced Spectroscopic Portal Program. I would also like to thank 
the Committee for its support of DNDO’s mission to reduce the risk 
of radiological and nuclear terrorism to the Nation. 

Over the past three years, we have made substantial invest- 
ments in the development of the next generation of radiation portal 
monitor known as the Advanced Spectroscopic Portal, or ASP. The 
goal of the ASP program is to advance passive detection for cargo 
inspection. ASP is expected to automatically discriminate threat 
from non-threat materials for unshielded to lightly shielded threats 
in primary inspection while improving identification capability in 
secondary inspection. To ensure that ASP systems achieve the nec- 
essary technical and operational performance, we are in the midst 
of putting them through a rigorous test campaign. ASP has already 
advanced through several rounds of performance testing, and will 
face field validation at ports of entry, as well as independent oper- 
ational testing and evaluation conducted by the DHS Science and 
Technology Directorate’s operational test authority. The successful 
completion of these tests, along with other analyses in consultation 
with the National Academy of Sciences, will then inform the Sec- 
retary’s certification decisions for ASP this fall. 

In addition, during the execution of our test campaigns, we have 
cooperated with the Government Accountability Office and the Na- 
tional Academy of Sciences to provide both groups with information 
and visibility into our testing and analysis processes. The resulting 
review as issued by the GAO and the NAS provide a valuable ex- 
ternal assessment of the ASP program. 

The 2008-2009 set of ongoing tests reflect a number of steps 
DNDO has taken to reform test processes. For example, one of our 
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most fundamental improvements has been to standardize test 
event planning. We now utilize detailed instructions with a six- 
milestone process to ensure that test planning is done openly, that 
partner inputs are included, and that test events are designed to 
provide the data required to meet the objectives of the test and to 
help make programmatic decisions. Additionally, some improve- 
ments in the ASP program have been the result of DHS-wide en- 
hancements to program management. The changes put in place 
within DHS further ensure that any eventual certification or acqui- 
sition decisions are consistent with DHS priorities and made with 
a strong acquisition management foundation. 

Specifically, we are implementing DHS’s new management direc- 
tive 102-01 for large acquisition programs which requires a com- 
plete set of analysis, testing and documentation as input for an ac- 
quisition decision. This set includes, for example, analysis of alter- 
natives, concept of operations, operational requirements, mission 
need statements, cost-benefit analysis and others. We are acutely 
aware that we must integrate end-user requirements and conduct 
tests and evaluation campaigns. We recognize that the develop- 
ment and deployment of new systems can be expensive and the 
cost to taxpayers must be justified by the increased capabilities of 
the equipment. We feel that the plans and procedures now in place 
for the ASP program provide a sound foundation for future certifi- 
cation and acquisition decisions. ASP systems have been under re- 
view and evaluation for over three years, and further improve- 
ments will always be possible, but I believe that after the plan test- 
ing and analysis is complete and the requirements of MD 102-01 
have been fulfilled, DHS will be in a position to make an informed 
decision. 

As a final note, I would like to emphasize that while we are dili- 
gently working to characterize and evaluate the performance of 
ASP to ensure that operators receive appropriate equipment for 
their mission, the ASP program is only one piece of the multi-lay- 
ered solution we call the global nuclear detection architecture 
through which we seek to integrate efforts across the government 
into an overarching strategy to improve the Nation’s nuclear detec- 
tion capabilities. Accordingly, we plan to continue deployments of 
detection systems at our official ports of entry while also dedicating 
increased time and effort to a wider range of pathways. In addition, 
we have been working to strengthen the architecture by filling 
gaps, improving technologies, building necessary infrastructure and 
raising awareness about radiological and nuclear threats and the 
role of detection systems. 

I look forward to continuing to work with our partners within 
DHS, other federal departments and State and local agencies and 
the Members of this subcommittee and the Congress to help keep 
the Nation safe from radiological and nuclear terrorism. 

This concludes my statement. I thank you for your attention. I 
am happy to answer any questions. 

[The prepared statement of Dr. Hagan follows:] 
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Prepared Statement of William K. Hagan 


Introduction: 

Good morning Chairman Miller, Ranking Member Broun, and distinguished Mem- 
bers of the Subcommittee. As Acting Deputy Director of the Domestic Nuclear De- 
tection Office ( DNDO) at the Department of Homeland Security (DHS ), I would like 
to thank the Committee for the opportunity to share lessons learned and progress 
to date on our Advanced Spectroscopic Portal (ASP) program. I would also like to 
thank the Committee for its support of DNDO’s mission to reduce the risk of radio- 
logical and nuclear terrorism to the Nation. 

DNDO was established to improve the Nation’s capability to detect and report at- 
tempts to import, possess, store, develop, or transport nuclear or radiological mate- 
rial for use against the Nation, and to further enhance this capability over time. 
To that end, our work is guided by our development of a global nuclear detection 
architecture (GNDA). DNDO has developed a time-phased, multi-layered, defense- 
in-depth GNDA that is predicated on the understanding that no single layer of de- 
fense can detect all radiological or and nuclear (rad/nuc) threats. For this reason, 
the GNDA provides multiple detection and interdiction opportunities overseas, at 
our borders, and within the United States to effectively increase the overall prob- 
ability of system success. DNDO has worked with intra- and interagency partners 
to develop time-phased strategies and plans for improving the probability of detect- 
ing and interdicting nuclear threats. DNDO will continue to enhance the GNDA 
over time by developing better detection technologies, working with our operational 
partners to improve concepts of operations (CONOPs), enabling real-time reporting 
of detection events, and supporting effective response to real threats. 

My testimony today will include a status update and lessons learned in DNDO’s 
efforts to address one aspect of the GNDA — scanning cargo containers at ports of 
entry. Specifically, I will focus on the ASP program — a program to improve the de- 
tectors used to perform this task. 

Role of Container Scanning and ASP in the GNDA: 

The United States border is the first layer within the GNDA where the United 
States has full control over detection and interdiction. As such, considerable effort 
and resources have been placed at this layer to provide comprehensive radiological 
and nuclear detection capabilities, particularly at ports of entry (POEs). After 9/11, 
considerable concern was raised about the possibility that terrorists could use the 
enormous volume of cargo flowing into the United States as a pathway for bringing 
in nuclear material or a nuclear weapon. By far, the largest mode for incoming 
cargo is maritime shipping containers, with approximately 11 million containers 
coming into the country every year. Additionally, in the Security and Accountability 
for Every ( SAFE ) Port Act of 2006, Congress mandated that all containers coming 
in through the top 22 ports, by volume, be scanned for radiation by the end of 2007. 

A key consideration in rad/nuc detection is the ability to effectively detect threats 
without impeding the flow of legitimate trade and travel across the border. United 
States Customs and Border Protection (CBP) currently scans cargo entering at our 
nation’s POEs using polyvinyl toluene (PVT)-based radiation portal monitors 
(RPMs) that can detect radiation, but cannot distinguish between threat materials 
and naturally occurring radioactive material (NORM), such as kitty litter and ce- 
ramic tiles. Narrowing down alarms to just those for dangerous materials is espe- 
cially important for POEs that have a high volume of containers, or those that see 
a high rate of NORM. To address this limitation, DNDO is developing next-genera- 
tion technology — the ASP program. The ASPs have shown significantly improved ca- 
pability to distinguish rad/nuc threats from non-threats over the hand-held instru- 
ments currently used in secondary screening. Thus, the introduction of ASP systems 
is expected to not only reduce the number of unnecessary referrals and false 
positives in primary but increase the probability of detecting dangerous materials 
in secondary. 

As you know, DNDO initiated the ASP program in 2006, building on previous 
work within CBP and the Science and Technology Directorate. ASP systems are the 
next generation of radiation portal monitors. ASP units are now being developed by 
two separate vendors. These units have been subjected to rigorous tests and both 
systems will complete several rounds of performance testing and field validation at 
POEs. Following these performance tests, both systems will complete independent 
operational testing and evaluation conducted by the DHS Science and Technology 
Directorate’s Operational Testing Authority. Test data will be analyzed and pro- 
vided in support of the Secretary’s Certification decision. DNDO is also engaged 
with the National Academy of Sciences (NAS) to allow them to review ASP testing 
and inform the certification process, as required in the FY 2008 and 2009 Homeland 
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Security Appropriations bills. Indeed, in its most recent report on ASP testing, the 
Government Accountability Office (GAO) acknowledged the many enhancements and 
lessons that DNDO has incorporated into its testing programs. 

Reviews to date and lessons learned: 

Since 2006, the ASP program has undergone extensive review from outside agen- 
cies, including the GAO, an Independent Review Team (IRT) established by the pre- 
vious DHS Secretary, and, most recently, the National Academy of Sciences. We 
have taken each review seriously, valued the recommendations that have been pro- 
vided, and, where we felt appropriate, we have incorporated their recommendations 
into the next stages of the program. 

The first reviews conducted by the GAO of the ASP program focused on testing 
conducted in 2005 as part of the original ASP vendor selection and the initial cost- 
benefit analysis used to evaluate potential deployment options for ASP systems. In 
its report, released in September 2006, GAO questioned the methods used by DNDO 
to quantify performance capabilities of new systems, and insisted that ASP perform- 
ance be evaluated against system requirements prior to full scale deployment. 
DNDO concurred with the need for additional testing prior to full scale deploy- 
ments, which were underway prior to the release of GAO’s report. These tests were 
conducted throughout 2007 and focused specifically on evaluating the performance 
of ASP systems in a number of testing environments. 

In September 2007, GAO recommended that DHS establish an independent body 
to conduct additional testing of ASP systems, which DNDO agreed to, launching the 
ASP-Independent Review Team (IRT), a team of independent experts, drawn from 
a wide range of institutions and backgrounds. The ASP-IRT Report, delivered in 
February 2008, provides a valuable independent assessment of the ASP program, 
and served as an important source of information, albeit based on the data that was 
available at the time. 

In addition to the reviews of 2007 testing, at the conclusion of initial “field valida- 
tion” testing in 2007, CBP identified a number of functional improvements, unre- 
lated to detector performance, that required modifications to ASP systems prior to 
deployment. DNDO postponed efforts to seek certification at that time, initiated new 
efforts to develop these requested changes , and conducted a new series of tests in 
2008 and early 2009 to ensure that these changes did not detract from detector per- 
formance. 

The 2008-2009 test campaign transitions the program from developmental to 
functionality and performance testing, culminating in full operational tests. This 
testing includes: 

1. System Qualification Testing, designed to demonstrate that ASP units are 
manufactured in accordance with processes and controls that meet the speci- 
fied design requirements; 

2. Performance Testing at the Nevada Test Site (NTS), designed to evaluate 
ASP, PVT, and radioisotope identification devices (RIID) detection and iden- 
tification performance against controlled, realistic threat materials, shielding 
and masking scenarios; 

3. Integration Testing, designed to determine whether the ASP systems are ca- 
pable of operating and interfacing with the other equipment found in oper- 
ational settings; 

4. Field Validation Testing, designed to exercise the ASP in a stream of com- 
merce environment at POEs; 

5. Operational Test and Evaluation (OT&E), designed to measure the oper- 
ational effectiveness and suitability of ASP. The OT&E will be independently 
conducted by the DHS Science and Technology Directorate (S&T). 

Recently, the GAO released its latest report, focusing on the testing conducted in 
2008 and 2009. GAO acknowledged improvements in ASP testing, but raised some 
concerns. DNDO agrees that analysis and review of test data is necessary. DNDO 
plans to continue study the results of testing as the ASP program progresses. How- 
ever, DNDO believes that the data analysis performed to date and the anticipated 
data from ongoing testing will be sufficient to inform an ASP certification decision 
in the future. 

In addition to the GAO’s reviews, DNDO and CBP have provided NAS with reg- 
ular testing updates. Recently, the NAS delivered an interim report to the Depart- 
ment and the Appropriations Committees. Like the NAS, DNDO sees the intrinsic 
value of continued testing and incremental deployment of ASP systems. However, 
DNDO must balance the need to better understand a complex system like ASP and 
the need to further reduce the risk of certain significant vulnerabilities and oper- 
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ational burdens. DNDO believes that the criteria outlined for certification are a suf- 
ficient threshold for determining when we have reached the point where deployment 
should begin. 

Ultimately, these reviews provide for a valuable external assessment of the ASP 
program. It is only through initial deployments that we will continue to learn more 
about the performance of these systems, and most rapidly bring about the improve- 
ments that are needed to address current limitations. ASP systems have been under 
review and evaluation for over three years now, and, while further improvements 
will always be possible, we should not delay the implementation of substantially im- 
proved capabilities. 

Acquisition lessons learned and changes made in response: 

DNDO has taken a number of steps to reform processes to ensure the success of 
ASP, as well as other development and acquisition programs. At the same time, 
these reforms are accompanied by a number of similar improvements to DHS-wide 
program management processes. 

With regard to testing, DNDO has taken a number of steps to improve internal 
procedures based on lessons learned from earlier tests. 

One of our most fundamental improvements has been through the standardized 
implementation of DNDO Operating Instruction 1, “Test Event Planning.” This de- 
tailed instruction lays out a six-milestone process that ensures that test planning 
is done openly, that partner inputs are included, and that test events are designed 
to provide the data required to meet the objectives of the test, and ultimately to 
help make programmatic decisions. 

We have also taken considerable steps to ensure that any ASP testing is respon- 
sive not just to DNDO requirements and objectives, but to all DHS partners. Prior 
to the 2008-2009 testing, DHS created a test planning working group that included 
DNDO, CBP, the DHS Operational Testing Authority (OTA), the Office of the Under 
Secretary for Management (USM), and the National Laboratories. Collectively, this 
group laid out the test campaign and assigned responsibilities for each test event 
to respective components. 

Finally, in the 2008-2009 ASP test campaign, based on lessons learned in the 
2007 test campaign series, we instituted strict entrance and exit criteria for each 
of the test events. These criteria were developed long before testing began, and were 
developed jointly with our operational partner, CBP. This has given us confidence 
that as the ASP systems have continued through this series of test events, require- 
ments are met prior to completion. 

DNDO has also made a number of program management changes based on les- 
sons learned throughout the program. In late 2008, DHS decided against exercising 
the next contract option for one of three ASP vendors. This decision reduced costs 
for carrying forward multiple, parallel development efforts. 

Finally, DNDO and the ASP program have benefited from a number of DHS-wide 
improvements to program management. DNDO has adopted Management Directive 
(MD) 102-01, which outlines the acquisition management process for DHS pro- 
grams. ASP program plans have been adapted to be consistent with the rigorous 
process outlined in the directive. This will further ensure that any eventual certifi- 
cation and acquisition decisions are consistent with DHS priorities and made with 
a strong acquisition management foundation. 

There is an important distinction between two key milestones for the ASP pro- 
gram — Secretarial certification and the MD 102-01 milestone decision for purchase 
and deployment of ASP. The former is a rather loosely defined milestone, which we 
have clarified by defining a “significant increase in operational effectiveness,” re- 
quiring that the Secretary formally state that she believes the ASP system is “better 
than” the current system. The latter is a very well defined milestone that was devel- 
oped for all large programs within the Department of Homeland Security. Items 
such as mission needs, operational requirements, analysis of alternatives, etc., are 
part of the MD 102-01 process. No ASP production units can be purchased and de- 
ployed without successfully navigating the MD 102-01 process. The Secretarial cer- 
tification requirement is in addition to and in advance of the MD 102-01 deploy- 
ment decision. 

Together, the lessons learned in testing and program management, along with the 
introduction and adherence to MD 102-01 have significantly improved the ASP pro- 
gram. At the same time, it is important to note that these lessons learned have not 
only benefited the ASP program; they are being applied across DHS. 
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ASP Path Forward: 

The plans and procedures in place for the ASP program provide a sound path for- 
ward for ASP certification and future acquisition decisions. The current path to cer- 
tification includes testing, accompanied by the analysis of results, to ensure that 
Secretary Napolitano has sufficient information for ASP certification. ASP systems 
have been under review and evaluation for over three years now, and, while further 
improvements will always be possible, a certification decision will determine wheth- 
er or not the systems address increase the probability of detecting dangerous mate- 
rials while minimizing the operational burdens. 

ASP was designed to improve capabilities in both primary and secondary inspec- 
tions. For primary applications, we have defined a “significant increase in oper- 
ational effectiveness” as being a quantified reduction in unwanted referrals to sec- 
ondary inspection, while maintaining similar or better sensitivity in detecting mate- 
rials of concern. Ultimately, the degree to which ASP systems meet this objective 
is driven by sensitivity thresholds at which the systems will be operated, similar 
to the threshold used for current PVT systems. Therefore, the advantage in primary 
can be viewed as an improvement in efficiency of operations (less unwanted refer- 
rals), while maintaining the same or better detection efficiency. The relative degree 
to which we realize these benefits with respect to the current systems will vary by 
port, depending on the operational thresholds at which the current PVT systems are 
set. 

The evaluation of ASP systems in secondary inspection is more direct. The ability 
of ASP systems to identify and resolve the source of radiation is directly compared 
to the ability of current capabilities to perform the same functions. In this instance, 
a “significant increase in operational effectiveness” is defined as a quantified im- 
provement in the ability to identify materials of concern. 

Both of these definitions were developed through coordination between DNDO, 
CBP, and USM. This definition has been approved by DHS senior leaders and has 
served as the guide for developing test campaigns to meet these test objectives. 
Again, because of the rigor that has gone in to developing and quantifying these im- 
provements, we are seeking to remove ambiguity in the evaluation process, and en- 
sure that any certification and acquisition decisions are made consistent with DHS 
priorities and objectives. 

Conclusion: 

DNDO will continue to work with CBP and other partners within and beyond 
DHS to improve the Nation’s ability to detect radiological and nuclear threats at our 
ports and borders. DHS is facing a challenge at our ports and borders as the De- 
partment balances facilitating the flow of goods and commerce with the need to suf- 
ficiently scan cargo for radiological or nuclear threats as it enters our nation. As 
both the President and Secretary have said, the Nation will need more technology 
to meet its security challenges and the technologies that DNDO is pursuing, of 
which ASP is but one example, are a critical component in addressing that chal- 
lenge. 

Our efforts to develop and evaluate ASP systems are sound. Current test results 
are capturing the benefits of ASP systems, and the reviews to date have provided 
a valuable assessment of the program and identified a number of key lessons 
learned. 

I welcome and appreciate the Committee’s active engagement with this program, 
and look forward to continuing our cooperation as we move forward together. Chair- 
man Miller, Ranking Member Broun, and Members of the Subcommittee, I thank 
you for your attention and will be happy to answer any questions that you may 
have. 


Biography for William K. Hagan 

Dr. William Hagan serves as Acting Deputy Director of the Department of Home- 
land Security’s (DHS) Domestic Nuclear Detection Office (DNDO). Prior to that, he 
was Assistant Director for the Transformational Research and Development (R&D) 
Directorate at DNDO. In that role Dr. Hagan was responsible for long-term R&D 
seeking technologies that can make a significant or dramatic positive impact on the 
performance, cost, or operational burden of detection components and systems. 

Prior to DNDO, he was a Senior Vice President at Science Applications Inter- 
national Corporation (SAIC). Business areas included nuclear technology (analysis, 
detection, and applications), telecommunications, optics, transportation, system inte- 
gration, and technology assessments during his thirty years at SAIC. 
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Dr. Hagan earned a Bachelor of Science in Engineering Physics in 1974, Master 
of Science in Physics in 1975, and Master of Science in Nuclear Engineering in 1977 
from the University of Illinois at Urbana. He received his Ph.D. in Physics from the 
University of California-San Diego in 1986. He holds three patents. Dr. Hagan was 
appointed to Senior Executive Service in 2006. 

Chairman Miller. Thank you, Dr. Hagan. 

Mr. Owen for five minutes. 

STATEMENT OF MR. TODD C. OWEN, ACTING DEPUTY ASSIST- 
ANT COMMISSIONER, OFFICE OF FIELD OPERATIONS, U.S. 

CUSTOMS AND BORDER PROTECTION, DEPARTMENT OF 

HOMELAND SECURITY 

Mr. Owen. Good morning, Chairman Miller, Ranking Member 
Broun, distinguished Members of the Subcommittee. As the Acting 
Deputy Assistant Commissioner for the Office of Field Operations 
with U.S. Customs and Border Protection, I am honored to be here 
this morning alongside Dr. Hagan to discuss the detection of radio- 
active and nuclear material in cargo containers and the future role 
that Advanced Spectroscopic Portal technology will have on CBP 
operations. 

I would also like to express my gratitude to the Congress for its 
continued support of CBP initiatives. Among the numerous prior- 
ities that were recognized in the American Recovery and Reinvest- 
ment Act, Congress provided CBP with $100 million worth of stim- 
ulus funding towards non-intrusive inspection equipment. This 
funding will allow CBP to upgrade and expand its successful Non- 
Intrusive Inspection (Nil) program and more efficiently inspect 
containers and vehicles crossing the border, allowing them to enter 
our country and its commerce in a safe and prompt manner. 

CBP has made tremendous progress in ensuring that supply 
chains importing goods into the United States are more secure 
against potential exploitation by terrorist groups aiming to deliver 
weapons of mass effect. CBP uses a multi-layered approach to en- 
sure the integrity of supply chains from point of stuffing through 
the arrival in the U.S. ports of entry. This multi-layered defense 
is built upon interrelated initiatives, which include the 24-hour 
rule in the Trade Act of 2002, the automated targeting system, the 
use of Non-intrusive Inspection equipment and radiation portal 
monitors, our container security initiative and the Customs Trade 
Partnership Against Terrorism program. These complementary lay- 
ers enhance security and protect our nation. 

Prior to 9/11, not a single radiation portal monitor and only 64 
large-scale non-intrusive inspection systems were deployed to our 
nation’s borders. By October of 2002, CBP had deployed the first 
RPM at the Ambassador Bridge in Detroit. Today CBP has well 
over 1,200 RPMs and 227 large-scale Nil technology systems de- 
ployed nationwide. Nil technology allows the officers to detect pos- 
sible anomalies, anomalies which may indicate the presence of a 
weapon of mass effect or some other contraband, and to date in fis- 
cal year 2009, CBP Nil systems have conducted over 3.1 million 
exams resulting in over 7,800 narcotic seizures with a total weight 
of over 2.6 million pounds as well as $6.2 million in currency 
seized. 

In addition to the significant strides made in the area of Nil 
equipment, CBP also continues to deploy first-generation radiation 
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portal monitors to our ports of entry. Currently, 97 percent of the 
trucks and 93 percent of the personally owned vehicles arriving 
from Canada, 100 percent of the trucks and vehicles from Mexico 
and 98 percent of the arriving sea containers are scanned by our 
current radiation detection technologies. In total, CBP scans 98 
percent of all cargo arriving into the United States by land and sea 
using radiation portal monitors. In addition, CBP officers scan 100 
percent of general aviation aircraft arriving to the United States 
from foreign destinations using hand-held radiation identification 
devices. 

Since the first RPM was deployed in 2002, CBP officers have 
scanned over 368 million conveyances for the presence of radiation, 
and we have resolved 2.1 million radiological alarms successfully 
with minimal or no impact to the flow of legitimate trade and trav- 
el. CBP continues to closely coordinate with key stakeholders to en- 
sure the impact of this activity causes minimal disruption to port 
operations. The first-generation RPM systems, although very sen- 
sitive, do have limitations. While they alert CBP officers to the 
presence of radiation, a secondary exam is necessary to positively 
identify the specific isotope causing the alert. In the event that a 
CBP officer is unable to positively resolve the alert, scientific reach- 
back is available 24 hours a day, seven days a week. 

The ASP is expected to enhance our detection capability while 
significantly reducing the number of secondary examinations. This 
is due to its ability to distinguish between actual threats and nat- 
ural or medical radiation sources that are not security threats. 
CBP has worked closely with the DNDO in the development and 
operational testing of ASP, has provided DNDO with functional re- 
quirements and has been actively engaged in every step of the eval- 
uation process. CBP’s focus for operational testing is to evaluate 
the effectiveness for systems deployed in our operational environ- 
ments. We will continue to work with DNDO and the DHS Science 
and Technology Directorate towards secretarial certification, and 
we are also working within DHS to ensure that any future ASP ac- 
quisitions and deployment decisions are consistent with DHS prior- 
ities. The decision to purchase and deploy ASPs in the operational 
arena will be based on CBP’s mission needs, operational require- 
ments, comprehensive cost-benefit analysis to include the full un- 
derstanding of maintenance and operation costs, and analysis alter- 
natives and other considerations. 

Mr. Chairman, Members of the Subcommittee, today I have ad- 
dressed CBP’s commitment to invest in new technologies and 
emerging technology aimed at enhancing cargo security. We must 
continue to maintain our tactical edge by integrating new tech- 
nology into our ports of entry. 

Thank you again for the opportunity to be here this morning and 
I would be happy to answer any questions. 

[The prepared statement of Mr. Owen follows:] 

Prepared Statement of Todd C. Owen 

Chairman Miller, Ranking Member Broun, esteemed Members of the Sub- 
committee, it is a privilege and an honor to appear before you today to discuss the 
work of U.S. Customs and Border Protection (CBP), particularly the detection of ra- 
dioactive and nuclear material in cargo containers and the future role that the Ad- 
vanced Spectroscopic Portal (ASP) program will have on our operations. CBP strives 
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to continually improve the security of cargo entering our borders and facilitate the 
flow of legitimate trade and travel. Included in this process, over 98 percent of all 
arriving maritime containerized cargo is presently scanned for radiation through ra- 
diation portal monitors. 

I want to begin by expressing my continuing gratitude to Congress for its contin- 
ued support for the mission and people of CBP. It is clear that the Congress is com- 
mitted to providing CBP the resources we need in order to increase and maintain 
the security of our borders. We appreciate your efforts and assistance. 

CBP is the largest uniformed, federal law enforcement agency in the country. We 
station over 20,000 CBP officers at access points around the Nation, including at 
air, land, and sea ports. As of mid-May, we have deployed over 19,000 Border Patrol 
agents between the ports of entry. These forces are supplemented with 1,058 Air 
and Marine agents, 2,318 agricultural specialists, and other professionals. These 
personnel are key players in the implementation of Secretary Napolitano’s South- 
west Border Security Initiative. 

CBP continues to execute all of its responsibilities, which include stemming the 
illegal flow of drugs, contraband and people, protecting our agricultural and eco- 
nomic interests from harmful pests and diseases, protecting American businesses 
from theft of their intellectual property, enforcing textile agreements, tracking im- 
port safety violations, regulating and facilitating international trade, collecting im- 
port duties, facilitating legitimate travel, and enforcing United States trade laws. 
CBP facilitates lawful immigration, welcoming visitors and new immigrants, while 
making certain those entering this country are indeed admissible and taking appro- 
priate action when an individual fears being persecuted or tortured if returned to 
their home country. At the same time, our employees maintain a vigilant watch for 
terrorist threats. In FY 2008, CBP processed more than 396 million pedestrians and 
passengers, 122 million conveyances, 29 million trade entries, examined 5.6 million 
sea, rail, and truck containers, performed over 25 million agriculture inspections, 
apprehended over 720 thousand illegal aliens between our ports of entry, encoun- 
tered over 220 thousand inadmissible aliens at the ports of entry, and seized more 
than 2.8 million pounds of illegal drugs. 

We must perform our important security and trade enforcement work without sti- 
fling the flow of legitimate trade and travel that is so important to our nation’s 
economy. These are our twin goals: border security and facilitation of legitimate 
trade and travel. 

CBP OVERVIEW 

I am pleased to appear before the Subcommittee today to highlight key accom- 
plishments related to container security, particularly those related to new and 
emerging technology. CBP has made tremendous progress in securing the supply 
chains bringing goods into the United States from around the world to prevent their 
potential use by terrorist groups that seek to deliver weapons of mass effect. The 
use of cutting-edge technology has greatly increased the ability of front line CBP 
Officers to successfully detect and interdict illicit importations of nuclear and radio- 
logical materials. CBP uses a multi-layered approach to ensure the integrity of the 
supply chain from the point of stuffing through arrival at a U.S. port of entry. This 
multi-layered approach includes: 

• Advanced Information 

o 24-Hour Rule 

o Automated Targeting Systems 
o Importer Security Filing 

• The Customs Trade Partnership Against Terrorism 

• The Container Security Initiative 

• The Secure Freight Initiative 

• Use of Non-Intrusive Inspection Technology and Mandatory Exams for All 
High-Risk Shipments 

I will discuss each one of these layers in greater detail with particular focus on 
our radiation and nuclear detection capabilities. 

ADVANCE INFORMATION 

CBP requires advanced electronic cargo information as mandated in the Trade Act 
of 2002 (including the 24-hour rule for maritime cargo). Advanced cargo information 
on all inbound shipments for all modes of transportation is effectively evaluated 
using the Automated Targeting System (ATS) before arrival in the United States. 
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ATS provides decision support functionality for CBP officers working in Advanced 
Targeting Units (ATUs) at our ports of entry and CSI ports. The system provides 
uniform review of cargo shipments for identification of the highest threat shipments, 
and presents data in a comprehensive, flexible format to address specific intelligence 
threats and trends. ATS uses a rules-based program to highlight potential risk, pat- 
terns, and targets. Through rules, the ATS alerts the user to data that meets or 
exceeds certain pre-defined criteria. National targeting rule sets have been imple- 
mented in ATS to provide threshold targeting for national security risks for all 
modes: sea, truck, rail, and air. 

The Importer Security Filing interim final rule, also known as “10 + 2,” went into 
effect earlier this year and has already yielded some promising results. This pro- 
gram will provide CBP timely information about cargo shipments that will enhance 
our ability to detect and interdict high risk shipments. Comments on aspects of this 
rule were accepted until June 1, 2009, and implementation using informed compli- 
ance will continue until January of next year. Shipments determined by CBP to be 
high-risk are examined either overseas as part of our Container Security Initiative 
or upon arrival at a U.S. port. 

Customs Trade Partnership Against Terrorism 

The Customs Trade Partnership Against Terrorism (C-TPAT) is an integral part 
of the CBP multi-layered strategy, in that CBP works in partnership with the trade 
community to better secure goods moving through the international supply chain. 
C-TPAT has enabled CBP to leverage supply chain security throughout inter- 
national locations where CBP has no regulatory reach. In 2009, CBP will continue 
to expand and strengthen the C-TPAT program and ensure that certified member 
companies are fulfilling their commitment to the program by securing their goods 
moving across the international supply chain to the United States. To carry-out this 
critical tenet of C-TPAT in 2009, teams of Supply Chain Security Specialists (SCSS) 
will conduct validations and revalidations of C-TPAT members’ supply chains to en- 
sure security protocols are reliable, accurate, and effective. 

As C-TPAT has evolved, we have steadily added to the rigor of the program. CBP 
has strengthened the C-TPAT program by clearly defining the minimum-security 
requirements for all categories of participants wishing to participate in the program 
and thereby gain trade facilitation benefits. As of June 18, 2009, there are 9,286 
companies certified into the C-TPAT program. CBP’s goal is to validate all partners 
within one year of certification, revalidate all companies not less than once every 
three years and revalidate all U.S./Mexico highway carriers on an annual basis, 
based on the risk associated with the Southern Border Highway Carrier sector of 
C-TPAT. 

Container Security Initiative 

To prevent terrorists and their weapons from entering the United States, CBP has 
also partnered with other countries through our Container Security Initiative (CSI). 
In FY 2008 CBP Officers stationed at CSI ports reviewed over 11 million bills of 
lading and conducted over 74,000 exams in conjunctions with their host country 
counterparts. Because of the sheer volume of sea container traffic, containerized 
shipping is uniquely vulnerable to terrorist exploitation. Under CSI, which is the 
first program of its kind, we are partnering with foreign governments to identify 
and inspect high-risk cargo containers at foreign ports before they are shipped to 
our seaports and pose a threat to the United States and to global trade. 

CBP Officers stationed at foreign CSI ports review 100 percent of the manifests 
originating and/or transiting those foreign ports for containers that are destined for 
the United States. In locations where the tremendous volume of bills prevents the 
CSI team at the port itself from performing 100 percent review, or during port shut- 
downs, CSI targeters at the National Targeting Center provide additional support 
to ensure that 100 percent review is accomplished. Utilizing the overseas CSI team 
and the CSI targeters at our National Targeting Center, CBP is able to achieve 100 
percent manifest review for the CSI program. 

Today, CSI is operational in 68 ports covering 86 percent of the maritime contain- 
erized cargo shipped to the United States. 

Secure Freight Initiative 

The Secure Freight Initiative (SFI) is an unprecedented effort to build upon exist- 
ing port security measures by enhancing the United States Government’s ability to 
scan containers for nuclear and radiological materials in seaports worldwide and to 
better assess the risk of inbound containers. Secure Freight will provide carriers of 
maritime containerized cargo with greater confidence in the security of the shipment 
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they are transporting, and it will increase the likelihood of an uninterrupted and 
secure flow of commerce. This initiative is the culmination of our work with other 
government agencies, foreign governments, the trade community, and vendors of 
leading edge technology. 

Moving forward, CBP will prioritize future deployments of scanning systems to lo- 
cations of strategic importance by identifying seaports where non-intrusive imaging 
and radiation detection data would be most practical and effective in deterring the 
movement of weapons of mass destruction via containerized cargo. Under this strat- 
egy, the additional scan data provided by SFI will enhance DHS’ risk-based and lay- 
ered approach to securing maritime containerized cargo. We will continue to work 
with Congress to enhance the safety of our nation’s ports and the security of incom- 
ing cargo. 

Non-intrusive Inspection/Radiation Detection Technology 

Today I will specifically address large-scale X-ray and gamma imaging systems 
and radiation detection devices; technologies that play a critical role in our layered 
enforcement strategy. 

The deployment of imaging systems and radiation detection equipment has con- 
tributed to CBP’s tremendous progress in ensuring that supply chains bringing 
goods into the United States from around the world are secure against exploitation 
by terrorist groups that seek to deliver weapons of mass effect. 

Non-intrusive Inspection (Nil) technology serves as a force multiplier that allows 
officers to detect possible anomalies between the contents of the container and the 
manifest. CBP relies heavily on the use of Nil as it allows us to work smarter and 
more efficiently in recognizing potential threats. 

Prior to 9/11, not a single Radiation Portal Monitor (RPM), and only 64 large-scale 
Nil systems, were deployed to our nation’s borders. By October of 2002, CBP had 
deployed the first RPM at the Ambassador Bridge in Detroit. Today, CBP has de- 
ployed 1,250 operational RPMs at seaports, land border ports, and mail facilities, 
227 large-scale gamma ray or x-ray imaging systems and 3,000 small scale Nil sys- 
tems nationwide. Additionally, CBP has deployed over 1,382 Radiation Isotope Iden- 
tifier Devices (RIID) and over 17,542 Personal Radiation Detectors (PRD). These de- 
vices allow CBP to inspect 100 percent of all identified high-risk cargo. 

Currently, 97 percent of trucks and 93 percent of personally owned vehicles arriv- 
ing through northern border ports, 100 percent of vehicles arriving through south- 
ern border ports, and 98 percent of arriving sea containers are scanned by our radi- 
ation detection technologies. CBP uses RPMs to scan 98 percent of all cargo arriving 
in the U.S. by land and sea. In addition, CBP officers now use hand-held radiation 
identification devices to scan 100 percent of private aircraft arriving in the U.S. 
from foreign destinations. As of May 2009, CBP officers scanned over 368 million 
conveyances and successfully adjudicated 2.1 million radiological alarms. 

It is important to distinguish these deployments from the 100 percent mandate. 
These deployments refer to CBP’s domestic RPM deployments, which perform radi- 
ation detection (not imaging) of containers that are scanned in the U.S. but prior 
to release into the commerce. The 100 percent mandate requires scanning in a for- 
eign port and both imaging and radiation detection. 

The first generation RPM systems, although very sensitive, do have limitations. 
While they alert CBP officers to the presence of radiation, a secondary exam is nec- 
essary to positively identify the location and specific isotope causing the alert. In 
the event that a CBP officer is unable to positively resolve the alert, scientific reach 
back is available on a 24/7 basis through the National Targeting Center and CBP’s 
Laboratory & Scientific Services Division located in the northern Virginia area. 

Understanding these limitations and the need for more precise radiological detec- 
tion architecture, the DNDO was chartered to develop new technologies that will 
improve CBP’s radiation and nuclear detection capabilities. One of these new tech- 
nologies is the next generation RPM, or the Advanced Spectroscopic Portal (ASP). 

The ASP is able to distinguish between actual threats and natural or medical ra- 
diation sources that are not security threats. In doing so, the ASP is expected to 
enhance our detection capability, while significantly reducing the burden of respond- 
ing to the numerous benign, nuisance alarms that are mostly generated by everyday 
products. This will allow CBP to focus our staffing and resources on high-risk ship- 
ments and other border security initiatives. 

CBP COORDINATION WITH THE DOMESTIC NUCLEAR DETECTION OF- 
FICE (DNDO) 

In the course of our collaboration with DNDO, CBP brings knowledge of how our 
ports work, of the support needs of our front-line officers, and of the operational re- 
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quirements for new technologies that must work consistently in a broad array of en- 
vironments. Additionally, we must remain attuned to critical factors such as 
throughput and capacity as we seek to maintain an appropriate balance between se- 
curity and the facilitation of cross-border travel and trade. 

CBP has worked closely with DNDO in the developmental and operational testing 
of the ASP. A complete independent operational testing and evaluation will be con- 
ducted by the DHS S&T Director, T&E and Standards Director, Operational T&E, 
when the system is ready. CBP’s objective for operational testing is ensuring that 
systems are operationally acceptable and effective and can be deployed in our oper- 
ational environments. Specifically, CBP provided DNDO with functional require- 
ments for the ASP and has been actively engaged in every step of testing, including 
performance testing at the Nevada Test Site and Integration testing currently ongo- 
ing at a mock port of entry at the Pacific Northwest National Laboratory. 

During integration testing, CBP works closely with DNDO to assess each system’s 
performance as an integrated unit, including reach back capability and ancillary 
equipment such as traffic lights and automated gate arms that are essential to 
maintain positive control of vehicles at our congested ports of entry. In addition, 
CBP works with DNDO to assess and categorize each system’s defects to ascertain 
their technological impact on performance and their operational impact on front-line 
CBP officers — the users of the system. 

CBP will continue to work with DNDO towards Certification by the Secretary, 
which is dependent on demonstrating a “significant increase in operational effective- 
ness” over existing first generation radiation detection systems. Only after this Cer- 
tification has been reach can the discussion then turn to potential acquisition and 
deployments of the ASP systems. The decisions to purchase and deploy ASPs in the 
operational arena will be based on mission needs, operational requirements, and a 
full understanding of maintenance and operational costs, to include a comprehensive 
cost benefit analysis, an analysis of alternatives, etc. 

CONCLUSION 

In conclusion, I would like to say that technology plays an enormous role in secur- 
ing the supply chain. Security technology is continuously evolving, not only in terms 
of capability but also in terms of compatibility, standardization, and integration 
with information systems. It is important to note that there is no single techno- 
logical solution to improving supply chain security. As technology matures, it must 
be evaluated, and adjustments to operational plans must be made. Priority should 
be given to effective security solutions that complement and improve the business 
processes already in place, and which build a foundation for secure 21st century 
global trade. 

Mr. Chairman, Members of the Committee, today I have addressed CBP’s commit- 
ment to investing its efforts in the areas of new and emerging detection technology, 
as well as some of the steps we have taken towards enhancing cargo security. 

Thank you again for this opportunity to testify. I will be happy to answer any of 
your questions. 


Biography for Todd C. Owen 

Todd C. Owen is the Executive Director of the Cargo and Conveyance Security 
Office within U.S. Customs and Border Protection’s Office of Field Operations. As 
the Executive Director for the Cargo and Conveyance Security (CCS) Office since 
May 2006, Mr. Owen is directly responsible for all cargo security programs and poli- 
cies for CBP, including the “100 percent scanning initiative” announced in October, 
2007. Included within Mr. Owen’s responsibilities are the Container Security Initia- 
tive (CSI), the Customs-Trade Partnership Against Terrorism Program (C-TPAT) 
Office, all non-intrusive inspection technology and radiation portal monitor deploy- 
ments, the National Canine Enforcement Program, and the National Targeting Cen- 
ter, Cargo. 

Mr. Owen also coordinates CBP’s maritime cargo enforcement policies and activi- 
ties with the U.S. Coast Guard, and all air cargo efforts with the Transportation 
Security Administration. 

Previously, Mr. Owen was the Director of the C-TPAT program from January 
2005 through May 2006. During his tenure as C-TPAT Director, this 9,000 member 
strong industry partnership program was strengthened by more clearly defining the 
security measures which must be adopted for a member to be eligible to receive the 
trade facilitation benefits afforded by CBP. Strong management controls were imple- 
mented and hiring was increased, allowing for a significant increase in the level of 
foreign site assessments performed worldwide under this program. 


VerDate Oct 09 2002 


09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00052 Fmt 6633 Sfmt 6621 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



47 


Prior to arriving in Washington in January 2005, Mr. Owen was the CBP Area 
Port Director in New Orleans. As the Area Port Director, Mr. Owen was directly 
responsible for all CBP operations in New Orleans and throughout Louisiana. Two 
hundred forty officers are stationed in the seven Louisiana CBP port offices man- 
aged by the Area Director. 

Mr. Owen began his career as an Import Specialist in Cleveland, Ohio in 1990, 
and transferred to Miami in 1992. Through his eight years in South Florida, Mr. 
Owen held various trade related positions within Field Operations and the Office 
of Strategic Trade, before being selected for the New Orleans Area Port Director po- 
sition in 2000. 

Mr. Owen, a career member of the Senior Executive Service, is a graduate of John 
Carroll University in Cleveland, Ohio, and was a senior executive fellow at Harvard 
University’s John F. Kennedy School of Government. Mr. Owen also holds a Mas- 
ter’s degree in Public Administration from St. Thomas University in Miami, Florida. 


Discussion 

Chairman Miller. I now recognize myself for five minutes. Dr. 
Hagan, you have heard a lot of criticisms of this program this 
morning, that is the Maginot Line of terrorism. It is pretty unlikely 
that terrorists will actually try to go through this system when it 
would be relatively easy to go around it, that a lot of the testing 
seems to be an afterthought to justify a conclusion that was al- 
ready reached. The end-user, the Customs and Border Patrol, think 
that they are doing perfectly well and that their existing tech- 
nology could be improved upon relatively cheaply compared to the 
investment that would be required, the spending that would be re- 
quired for ASP. And both of the previous witnesses very much 
questioned the validity, the thoroughness of the testing that was 
being used to justify the decision to deploy ASP. What is the ur- 
gency? In view of all of that, what is the urgency about putting a 
certification decision before the DHS Secretary in October? 

Dr. Hagan. I guess there is a couple things I would like to say. 
First is that there is a sense of urgency regarding some of the limi- 
tations of the current system. We can’t talk about these in open 
session, but we would be happy to do that at a later time. But hav- 
ing said that, I don’t believe that today that there is a rush to de- 
ployment nor is there a ‘do it by October or something bad hap- 
pens.’ So what we are doing though is pushing forward as aggres- 
sively as we can to get to the decision point about deployment. In 
other words, we are not saying we have got to deploy these right 
now. We are saying we have got to get, as soon as possible, to the 
point where we make that decision, and as was said by the earlier 
witnesses, the information that is required to make that decision, 
as I mentioned in my oral statement, there is a lot of information, 
a lot of data, a lot of test results that have yet to be analyzed and 
brought together and integrated in this cost-benefit analysis, and 
that would be an important part of the decision that will be made 
in October. So, I guess I would disagree that we are rushing to de- 
ployment. I would say that we are rushing more towards getting 
all the information as fast as we can but doing it in a thoughtful, 
deliberate, disciplined way and through this MD 102-01 process 
that I mentioned, to get to that decision point. 

Chairman Miller. Mr. Owen, we heard again from GAO and the 
National Academies of Science that the work to this point did not 
really instill confidence in them that this new system would per- 
form as reliably, as effectively as we would like or even as the ex- 
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isting technology works. They seem not concerned about a delay. 
They in fact encouraged a delay. Do you agree with their rec- 
ommendation to stick with what we have got and not rush either 
to deploy or to certify or to decide to deploy? 

Mr. Owen. Sir, as the operator, the end-user of the systems, we, 
as front-line officers, need technology that will not only reduce the 
possibility of a missed threat but as well as reduce the rate of inno- 
cent alarms that our officers are spending their time on. In the 
port of Long Beach, for example, we have between 400 and 600 ra- 
diation alarms every single day. We have a team of about 100 offi- 
cers over the three shifts that work on just resolving these innocent 
alarms throughout the course of their workday. So having a new 
piece of technology or an enhancement to technology that will allow 
us to not only identify any missed potential threats, but reduce the 
number of innocent alarms, is something that we do support. But 
as I mentioned in our hearing, our testimony, we have been doing 
this for some time with the existing PVTs. We have been able to 
resolve those 2.1 million alarms without a negative throughput 
with the flow of commerce in our ports of entry. So the technology 
that we have is effective but again, as we have matured the tech- 
nology, we are looking for that technology which will give us an 
edge, if you will, a step up from what we have, particularly in 
terms of missing any threats that the PVTs may not capture right 
now. 

Chairman Miller. With respect to the existing technologies, the 
PVTs with the follow-up searches, are there enhancements, im- 
provements in that technology that seem available, achievable, and 
have those improvements, are those being pursued? 

Mr. Owen. Our current system with the PVT which will alert — 
the container will then go in a secondary where we will perform 
the analysis with the radiation isotope identifier and we push that 
information back to our laboratory and scientific folks — it is a proc- 
ess that we have crafted and we feel comfortable with. What we 
have seen in the operational area, in terms of the energy 
windowing that you heard from the first panel, is we have seen 
operational benefits to being able to make adjustments to the exist- 
ing PVTs to account for the recurring, burdensome, naturally oc- 
curring radioactive shipments that come through in legitimate 
cargo. So we have seen the benefits from energy windowing in the 
operational environment. 

Now, as to have we pushed those benefits as far as we can 
through energy windowing, we really need to defer to the sci- 
entists, to DNDO and our partners at Pacific Northwest National 
Laboratories to tell us if we have gone as far as we can with energy 
windowing. But again, as the operator of these systems, we have 
seen benefits in the ports of entry where we have deployed energy 
windowing. Have we maximized that? I would really defer to the 
scientists to give us that answer. 

Chairman Miller. I understand the principal problem with the 
current technology is as you have pointed out, not so much that it 
would miss — it is less a concern for missing radiation than the 
large number of false positives. Perhaps not false in the sense that 
it is not radiation, but it is not radiation that is a problem. How — 
I understand that the PVT system has been tested for that as well. 
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How did it fare compared to the existing — I am sorry. The ASP sys- 
tem, how did it fare compared to the PVT system in false positives? 

Dr. Hagan. I will take that. The actual test data is classified, but 
I can answer in a qualitative way, that in — there are two places 
in which the ASP is being tested and considered. One is in primary 
inspection and the other is secondary inspection. Do you want me 
to explain what I mean by that? 

Chairman Miller. Well, for the record at least, sure. 

Dr. Hagan. Okay. 

Chairman Miller. Briefly. 

Dr. Hagan. So a conveyance comes in and goes through primary 
inspection or screening, and goes through an RPM or radiation por- 
tal monitor of some sort. If that monitor alarms, then that convey- 
ance is referred to secondary inspection where it is further exam- 
ined either with another monitor, a radiation portal monitor or a 
hand-held RIID. And ASP is being considered for deployments to 
both of those, both primary inspection and secondary inspection. In 
the primary application, the ASP is a — has the ability to identify 
the type of — the source of the radiation. The PVT is not able to do 
that. So PVT simply says there is radiation, better go to secondary 
and off it goes. In the case of ASP, because of the spectroscopic na- 
ture of the system, it can discriminate between a threat material 
and non-threat material, or if it can’t discriminate, then it sends 
it to secondary as well. So it can dismiss and reduce the number 
of referrals to secondary. In the secondary application, if you have 
ASP there, then you are comparing essentially the performance of 
a large spectroscopic portal which is very, very large compared to 
a very small hand-held detector in which case the — again, I can’t 
talk about the specifics of the results but the ASP is far superior 
to the hand-held detector in terms of its ability to, in real time, 
identify the source of the radiation and dismiss it as appropriate. 

Chairman Miller. Is it classified to say which did better, the 
PVT or the ASP? 

Dr. Hagan. No. I am sorry. I should have said, yes. So the ASP 
does far better in the secondary application of being able to iden- 
tify — 

Chairman Miller. Well, which — I am sorry. Which system pro- 
duced more false positives, ASP or PVT? 

Dr. Hagan. In the most recent 

Chairman Miller. February. 

Dr. Hagan. Okay. In the most recent — you are talking about 
field validation testing, I think. 

Chairman Miller. Yes. 

Dr. Hagan. In field validation tests, the ASP system sent more — 
had more referrals to secondary than the PVT did. However, we 
have now — it is actually not surprising. It was the first time that 
ASP had been able to operate in a real operating environment, and 
so just as with PVT, adjustments were made to what are called 
thresholds in the ASP. So we are not making software changes, we 
are simply tuning the system as you do with the PVT, and then 
we have done a lot of analysis and simulation, as was mentioned 
earlier, to verify that these changes will in fact improve the per- 
formance and give us the expected performance. 
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Chairman Miller. Okay. It appears that my time has really and 
truly expired. 

Dr. Broun for five minutes. 

Mr. Broun. Thank you, Mr. Chairman. I don’t watch the clock 
very closely, as you very well know, and I will always give you — 
almost always give you a lot of leeway unless there is some par- 
ticular reason not to. 

But Dr. Hagan, what you are saying, it is my understanding that 
the ASPs — you just made a statement that they are better. It is my 
understanding that that is only with the lightly shielded radiation. 
Isn’t it true that with heavily shielded or moderately shielded radi- 
ation, that there is not much difference between the two programs? 
And isn’t it also true that if anybody is going to bring radiation- 
type special nuclear materials in any shipment, aren’t they going 
to be shielded and very highly so? 

Dr. Hagan. The answer to the second part first. One might spec- 
ulate that that would be the case but you can’t really know for 
sure, and so yes, it is probably likely that someone would shield it, 
but a knowledgeable adversary might not do that for other reasons 
which I will talk about later. But going back to the first part of 
your question, yes, the two systems, both PVT monitors and ASP 
monitors are what are called passive detection systems. They both 
suffer from the limitation that if a nuclear material or a weapon 
is shielded enough, then there just is no signal for either detector 
to detect. Now, so what we are — the long-range plan for this, and 
it is already being implemented in part, is to say that if a — by hav- 
ing these passive detectors, we are forcing the adversary, knowl- 
edgeable adversary, anyway, to heavily shield the object. Well, that 
heavily shielded object, then, is a very — well, I shouldn’t say very 
but is an easy target or an easy image or easy thing to identify in 
a radiographic image or non-intrusive inspection system which Mr. 
Owen talked about. So by having complementary, in the sense, or- 
thogonal type of systems, one that is passive system that will de- 
tect lightly shielded or unshielded material, and another system 
that will detect shielded material, then you have covered — you 
have done a good job of covering the complete spectrum of possibili- 
ties. Any system, of course, can be defeated by a knowledgeable ad- 
versary, but by having those two systems combined, then you really 
do a much better job of dealing with that problem. That being said, 
the amount of shielding that is needed, the effectiveness of PVT 
versus ASP for lightly or moderately shielded materials is — it de- 
pends on what it is you are shielding, and we would have to go into 
particulars and specifics about different types of objects, which we 
can’t do here. 

Mr. Broun. Thank you, Dr. Hagan. My time is about up. I will 
yield back. 

Chairman Miller. Ms. Dahlkemper for five minutes. 

Ms. Dahlkemper. Thank you, Mr. Chairman. 

I live on Lake Erie and so I go back and forth to Canada a fair 
amount and actually I felt much safer a few years ago when were 
pulled over. My father had had a stress test two weeks prior so I 
do actually have some personal experience with those innocent 
alarms. I didn’t understand what was going on at the time. It was 
the first time I had ever been pulled over. 
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Mr. Owen. I hope we didn’t cause too much stress. 

Ms. Dahlkemper. No, no, we were fine, but I actually felt much 
better coming back home and telling people who had no idea that 
this was actually being detected as we all would drive back home 
from Canada. But DHS has spent almost half a million — a billion 
dollars deploying PVT monitors at border crossings and ports 
across the country, including the one I came across, obviously. 
Does, Mr. Owen, the Customs and Border Protection believe that 
the current system, using PVTs and doing the secondary inspection 
with hand-held detectors and other means, is working well in 
terms of keeping us safe and moving commerce along? And I guess 
I am wondering what is the cost of manpower versus what is the 
cost of ASP? You know, I am looking at the dollars spent here, 
knowing we have limited resources and where are we best spend- 
ing our money. 

Mr. Owen. And I would just respond, ma’am, that again two 
parts to that equation is, is the PVT finding all of the threats or 
are there threats that could be slipping through just because of the 
physics involved, and that is where ASP can help us in that regard. 
The second piece of that would be, can we respond to the number 
of secondaries that are caused by the existing PVT systems, and 
the answer is yes, we demonstrate that. There is a resource impact 
on this. Again, we would look at Los Angeles, we look at what we 
do up in the Peace Bridge there in Buffalo where you probably 
came through. It is a timely process. The officers do have to go 
through each secondary exam and treat it as if it is a real threat. 
We cannot let our guard down and just assume it was your father- 
in-law, that he had medical testing, let you go down. We have to 
take you out of the vehicle, we have to go through that whole proc- 
ess. So having a system that reduces the number of innocent 
alarms that are sent into secondary will help us do that. As to is 
the cost of the manpower savings offset by the ASP, I think that 
is some of the work that the DNDO is doing with their overall cost- 
benefit analysis. It will take into consideration those variables. 

Ms. Dahlkemper. I wanted to ask you too about the mainte- 
nance and operation costs of the ASPs and they are estimated to 
be anywhere from five to twelve times more expensive than the 
costs of running the PVTs, and so in the worst case ASPs will cost 
about $100,000 a year to run as compared to $8,000 a year for the 
PVTs. Does Customs have the budget to support this kind of sys- 
tem if it were deployed nationally, and have you thought about 
what you would have to give up in terms of personnel or other, you 
know, equipment to run this? 

Mr. Owen. Yes, and that is a very good question. That is a con- 
cern that we do have now. The ASP systems, depending on how the 
final outcome of the cost-benefit and the cost of the systems, I 
think it is well accepted that it is going to cost more than what we 
have with the PVTs. Currently we have about 1,250 PVTs deployed 
nationwide. We continue to expand along the northern border. We 
are going to get — up to 100 percent of the cargo coming in from 
Canada this year will be covered by PVTs. We will end up with 
about 1,500 PVTs by the end of this calendar year. There is an ex- 
tensive operation and maintenance tail that comes with that. I 
think any deployment decisions going forward with ASPs needs to 
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be cognizant of the impact that will have on the operator. There 
is much talk about the initial acquisition cost and the deployment 
cost to buy it, put it in the ground, but I don’t think we can forget 
about the operation and maintenance tail that comes along with 
this, that will then fall on the backs of the operator, in this case 
CBP. So it is something that we are concerned with. It is some- 
thing that again is going into the overall cost adjustments for what 
we can expect with the ASP systems. 

Ms. Dahlkemper. Do you have any thoughts on what this would 
do in terms of personnel? Would it free up great numbers for other 
purposes? 

Mr. Owen. Well, my view on this is, when you look at the way 
the layouts are in the seaports, you have to have still an officer 
man those exit gates where you have the radiation portal monitors. 
So whether it is manned with two officers or one officer because the 
alarms are less frequent, there will still be a cost figure associated 
with the manpower. There may be some operational adjustments 
that we can make in the secondary areas but I think the cost sav- 
ings are something that we still need to measure once we are con- 
fident that the ASPs are delivering as we hope they will do. So 
there will be some savings from that. Will that offset the increase 
in operation maintenance costs associated with the ASP? That is 
something that we have to take very careful consideration of. 

Ms. Dahlkemper. Thank you. My time is up. I yield back. 

Chairman Miller. Thank you, Ms. Dahlkemper. That will be the 
last round of questions. Dr. Hagan, I trust that when DHS has 
completed its cost-benefit analysis, that we will get a copy that is 
still warm from the printer. 

Dr. Hagan. Okay. 

Chairman Miller. So under the rules — before we bring the hear- 
ing to a close, I do want to thank all of our witnesses for testifying 
before the Subcommittee today. Under the rules of the Committee, 
the record will remain open for two weeks for additional state- 
ments from the Members and for answers to any follow-up ques- 
tions any Member of the Committee may have for witnesses, and 
I understand that Dr. Broun does have questions. The witnesses 
are excused and the hearing is now adjourned. 

[Whereupon, at 11:41 a.m., the Subcommittee was adjourned.] 
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Preventing radioactive material from being smuggled into the United 
States is a key national security objective. In particular, terrorists could 
use special nuclear material such as highly enriched uranium (HEU) or 
plutonium in a nuclear weapon; other radioactive materials could be used 
in a radiological dispersal device (a “dirty bomb”). The national security 
mission of U.S. Customs and Border Protection (CBP), an agency within 
the Department of Homeland Security (DHS), includes screening for 
smuggled nuclear or radiological material while facilitating the flow of 
legitimate trade and travel. To screen cargo at ports of entry. CBP 
conducts primary inspections with radiation detection equipment called 
portal monitors — large stationary detectors through which cargo 
containers and vehicles pass as they enter the United States. When 
radiation is detected. CBP conducts secondary inspections using a second 
portal monitor to confirm the original alarm and a handheld radioactive 
isotope identification device to identify the radiation’s source and 
determine whether it constitutes a threat. CBP officers must investigate 
each alarm until they are convinced that the vehicle, occupants, and any 
cargo pose no t hreat and can be allowed to enter the United States. 

According to DHS’s Domestic Nuclear Detection Office (DNDO), the 
current generation of radiation detection equipment has limitations. 1 
Specifically, the poly vinyl toluene (PVT) portal monitors currently in use 
can detect radiation but cannot identify the source. As a result, the 
monitors’ radiat ion alarms can be set off even by benign, naturally 
occurring radioactive material. One way to reduce the rate of such 
innocent alarms and thereby minimize unnecessary secondary 
inspections and enhance the flow of commerce— is to adjust the 
operational thresholds for the level of radiation required for PVTs to alarm 
(i.e., operate the PVTs at a reduced level of sensitivity). However, reducing 


'DNDO was established within DHS in 2005; its mission includes developing, testing, 
acquiring, and supporting the deployment of radiation detection equipment at U.S. ports of 
entry 1 ' CBP began deploying portal monitors in 2002, prior to DNDO's creation, under the 
radiation portal monitor prqject. For additional Information on DNDO's overall efforts to 
combat nuclear smugging, see GAO. Nuclear Detection: Domestic Nuclear Detection 
Office Should Improve Planning to Better Address Gaps and Vulnerabilities, GAO-09-257 
(Washington. D.C.: Jan. 29, 2009). 
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the sensitivity nifty make it more difficult to detect certain nuclear 
materials. 

Since 2005, DNDO has been developing ami testing advanced 
spectroscopic portals (ASP), a new type of portal monitor designed to 
both detect radiation and identify the source. The new portal monitors use 
technology similar to that in handheld identification devices currently 
used for secondary screening. Key differences from handheld 
identification devices include a larger number of detectors, more 
sophisticated soft ware, and a more extensive library of radiation 
signatures that may provide more consistent and rapid screening and may 
increase the likelihood of correct klentificafion. DNDO hopes to use the 
new portal monitors to replace at least some PVTs currently used for 
primary screening, as well as PVTs and handheld identification devices 
currently used for secondary screening. However, the new |>ortal monitors 
cost significantly more than PVTs. We estimated in September 2008 that 
the lifecycle cost of each standard cargo version of the ASP (including 
deployment costs) is about $822,000. compared with about $308,000 for 
the PVT standard cargo portal, ami that the total program cost for DNDO’s 
latest plan for deploying radiation portal monitors which relies on a 
combination of ASF’s and PVTs and does not deploy radiation portal 
monitors at all border crossings would be about $2 billion. 1 Moreover, 
CBP officials expect operation and maintenance costs to lx* significantly 
higher for ASF’s than for PVTs because of the great er complexity of ASP 
equipment. 

Concerned about the performance and cost of the new ASP monitors. 
Congress required the Secretary of Homeland Security to cert ify that the 
monitors will provide a “significant increase in operat ional effectiveness" 
before DNDO obligates hinds for hill-scale ASP procurement. 3 The 
Secretary must submit separate certifications for primary and secondary 
Inspection. In response, DNDO, CBP, and the DHS management 
directorate jointly issued criteria in July 2008 for determining whether the 
new technology provides a significant increase in operational 


"GAO, Combating Nuclear Smuggling: DOS's Program to Procure and Deploy Advanced 
Radiation Detection Portal Monitors Is Likely to Exceed the Department’s Previous Cost 
Estimates, GAO-08-1 108R (Washington, D.C.: Sept 22. 2008). 

^Consolidated Appropriations Act 2007, Pub. L No. 110-161, 121 Stat 1814. 2060 (2007); 
Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 2009. Pub. 
L No. 110-329, 121 StaL 3674.3679(2008). 
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effectiveness— four criteria for primary screening and two for secondary 
screening (see fig. 1). The primary screening criteria require that the new 
portal monitors detect |x>tential threats as well as or better Ilian PYTs, 
show improved performance in detection of HEU. and reduce innocent 
alarms. To meet the secondary screening criteria, the new portal monitors 
must reduce the probability of misidentiiying special nuclear material 
(e.g., HEU and plutonium) and the average time to conduct secondary 
screenings. 


Figure 1: DHS Criteria for Demonstrating a Significant Increase in Operational 
Effectiveness 


Primary screwring criteria 


When ipecial nuclear noienai Is present 
In cargo wthoii neturaty occirrng 
radioactive mdmil, Vie ASP pcobaMVy of 
a correct operational outcome musl be 
equal lo or greater than Ihal of Vie PVT 
iFor HEU ASPs musl show improvad 
performance compared lo PVTt el 
operation el Ihresnoldt I 

When special nuclear material Is present 
m cargo with nature#/ occurring radioac- 
tive material the ASP m plenary musl 
increase Via probabMy of a coned 
operational outcome compared lo Vie 
current end-to-end system 

Whan Kentabta medical or industrial 
Hoi opei are preierr in cargo Via ASP 
probabdty of a coned operational 
outcom# mud ba oqual lo or greattr than 
that of lha PVT 

Whan Vie only rattoedere source present 
m the cargo is naluraly occurring 
radio active material lha ASP mutt rotor at 
least 60 percent fewer conveyances tor 
further mspecfion man Vie PVT. 


Second ary screening criteria 



Saute OMS 


To demonstrate a significant increase in operational effectiveness for 
either primary or secondary screening. ASPs must sat isfy all of the criteria 
for that deployment opt ion, independent of satisfying the criteria for the 
other option. The criteria generally compare the new portal monitors to 
current-generation equipment as used under GBP's standard operating 
procedure. For example, the standard operating procedure for secondary 
screening calls for inconclusive readings to be sent for additional analysis 
to CBFs Laboratories and Scientific Services, which has access to 
additional software and trained experts. 
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DNDO designed and coordinated a series of tests, originally scheduled to 
run from April 2008 through September 2008, to determine whether the 
new portal monitors meet the certification criteria for primary and 
secondary screening and are ready for deployment. Key phases of testing 
completed to date Include verifying that ASPs meet DNDO’s performance 
specification, which was followed by concurrent testing of the new and 
current equipments' ability to detect and identify threats and of ASIV 
readiness to be integrated Into operations for both primary mid secondary 
screening at ports of entry. Two remaining phases not yet completed 
include field validation at four northern and southern border crossings and 
two seaports, as well as an independent evaluation, conducted by the DHS 
Science and Technology Directorate at one of the seaports, of the new 
|>ortal monitors' effectiveness and suitability (see fig. 2). Two ASP vendors 
have contracts with DNDO to develop the new portal monitors and are 
l>articipaiing in the round of testing that began in 2008. 4 DNDO designed 
the testing to allow each vendor’s system to complete all test phases and 
be certified based on its own performance as providing a significant 
increase in operational effectiveness. 


Figure 2: Test Sequence Leading Up to ASP Certification 
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and evaluation 



‘DNDO had a contract with a third ASP vendor whose system uses a mare expensive type 
of detector that must be cooled by liquid nitrogen. DNDO determined it was not in the best 
Interests of the government to exercise the option on the contract and allowed It to expire 
In November 2008, The vendors ASP did not participate in the 2008 round ot testing 
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We have raised concerns since 2006 regarding DNDO’s previous efforts to 
develop and lest the new porlal monitors. In October 2006, we found that 
DNDO’s analysis of the benefits and costs of deploying the new portal 
monitors relied on assumptions of their anticipated performance level 
instead of actual test data.' Among other things, we recommended that 
DNDO conduct further testing before s|>ending additional funds to 
purchase the new equipment. In September 2007, we testified that DNDO’s 
testing at the Department of Energy’s (DOE) Nevada Test Site did not 
represent an objective or rigorous assessment because DNDO used biased 
test methods that enhanced the apparent performance of the ASPs and did 
not test the limitations of their detection capabilities.' Most recently, we 
found in September 2008 that a DNDO report on testing conducted in 2007 
did not accurately depict test results and could potentially be misleading. 
We concluded that the results could identify areas for improvement but 
should not be used as indicators of ASPs’ overall performance. 

In this context, you asked us to review the 2008 round of testing leading up 
to the Secretary of Homeland Security's decision on ASP certification. We 
reviewed (1) the degree to which DBS’s criteria for a significant increase 
in operational effectiveness address the limitations of the current 
generation of radiation detection equipment, (2) the rigor of the testing as 
a basis for determining ASPs' of >e rational effectiveness and preliminary 
results of testing completed to dale, and (3) the extent to which the test 
schedule allows time for DHS to review and analyze results. This report 
updates our September 2008 testimony, which Included preliminary 
observations on the DHS criteria for a significant increase in operational 
effectiveness and the 2008 round of testing.* 


! GAO. Combating Nudear Smuggling: DNS's Cost-Benefit Analysis lo Support Die 
Purchase of New Radiation Detection Portal Monitors Was Not Based on Available 
Performance Data and Did Not Fully Evaluate All the Monitors ' Costs and Benefits, 
GAO-07- 133R (Washington, D.C.: Oct 17, 2006). 

‘GAO, Combating Nudear Smuggling: Additional Actions Needed to Ensure Adequate 
Testing of Next Generation Radiation Detection Equipment, GAO-07- 124 7T (Wasliington, 
D.C.: Sept 18. 2007). 

GAO, Combating Nuclear Smuggling: DBS's Phased Test Report on Advanced Portal 
Monitors Does Not Fully Disclose the Limitations of the Test Results. GAO-08-979 
(Washington, D.C.: Sept. SO, 2008). 

*GAO, Combating Nudear Smuggling: DHS Needs lo Consider the Full Costs and 
Complete AU Tests Prior to Making a Decision on Whether to Purchase Advanced Portal 
Monitors. GAO-08- 1 178T (Washington. D C.: Sept 26. 2006). 
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To conduct our review, we analyzed DHS’s criteria for a significant 
increase in operational effectiveness and DNDO’s written response to our 
detailed questions regarding the criteria Because the criteria compare the 
new portal monitors to existing equipment, we analyzed the threat 
guidance used to set detection thresholds for PVTs and interviewed DOE 
and national laboratory officials responsible for the guidance. In addition, 
we analyzed the test plans for the 2008 round of test ing, including the test 
schedule and reasons for any delays. We interviewed DNDO, CBP, and 
other DUS officials responsible for conducting and monitoring tests, and 
we observed 1 day each of performance testing at the Nevada Test Site 
and integration testing at DOE’s Pacific Northwest National Laboratory. 
We analyzed preliminary or final results for the phases of test ing 
completed during our review, and we interviewed DNDO and CBP officials 
regarding the results. (App. I presents a detailed discussion of the scope 
and methodology of our review.) 

We conducted this performance audit from May 2008 to May 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and perform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives.' 


*This report docs not include certain details about the capabilities and limitations of PVTs 
and ASPs that DHS considers to be ‘for official use only.’ We haw prepared a “for official 
use only* version of this report In which we Include such details (GAO-09-364SU) 
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DHS’s Criteria for 
Significant Increase in 
Operational 
Effectiveness Require 
a Marginal 
Improvement in the 
Detection of Certain 
Nuclear Materials and 
a Large Reduction in 
Innocent Alarms 


Although the DUS criteria for primary screening require an improved 
ability to detect certain nuclear materials at operational thresholds, ASPs 
could meet the criteria for improvement while still failing to detect 
anything more than lightly shielded material. DNDO officials acknowledge 
that passive radiation detection equipment, which includes both the new 
and current-generation portal monitors, is capable of detecting certain 
nuclear materials only when this material is unshiekled or lightly 
shielded. 10 For this reason, the DOE threat guidance used to set PVTs* 
detection threshold is based on the equipment’s limited sensitivity to 
anything more than lightly shielded nuclear material rather than on the 
assumption that smugglers would take effective shielding measures. DOE 
developed the guidance in 2002 and 2003 when CBP began deploying PVTs 
for primary screening. DOE and national laboratory officials responsible 
for the guidance told us the assumption of light shielding was based not on 
an analysis of the capabilities of potent ial smugglers to take effective 
shielding measures but rather on the limited sensitivity of PVTs to detect 
anything more than certain lightly shielded nuclear materials. In contrast, 
PVTs are more sensitive to the relatively strong radiation signature of 
other nuclear materials, and the threat guidance assumes a higher level of 
shielding for setting the operational threshold for detection of such 
materials. However, even for such materials, the DOE threat guidance 
assumes that shielding would not exceed a level provided by the contents 
of an average cargo container. 


Moreover, DNDO has not completed efforts to fine-tune PVTs’ software 
and thereby improve sensitivity to nuclear materials. As a result, the 
criteria compare ASPs to the current performance of PVTs and do not lake 
potential improvements into account, which affects any assessment of 
“significant” improvement over current technology. DNDO officials expect 
they can achieve small improvements to PVTs* performance through 
additional development of “energy windowing,” a technique currently 
being used in PVTs to provide greater sensitivity than otherwise possible. 
Pacific Northwest National Laboratoiy officials responsible for developing 
the technique also told us small improvements may be possible, and CBP 
officials have repeatedly urged DNDO to invest igate the potential of the 
technique. DNDO collected the data needed to further develop energy 


'"According to DNDO and CBP officials, active imaging techniques (e.g., radiography 
systems to provide images of the contents of cargo containers) and other measures 
complement radiation detection equipment. In particular, such measures jjrovide the 
capability to spot smuggled nuclear materials Uiat are too heavily shielded to be detected 
by PVTs or ASPs. 
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windowing during the 2008 performance testing at the Nevada Test Site 
but has not yet funded Pacific Northwest National Laboratory efforts to 
analyze the data and Anther develop the technique. 

Other aspects of the criteria for a significant increase in operational 
effectiveness require that ASPs either provide more than a marginal 
improvement in addressing other limitations of current-generation 
equipment or at least maintain the same level of performance in areas in 
which the current-generation equipment is considered adequate: 

• The primary screening requirement for an 80 percent reduction in the rate 
of innocent alarms could result in hundreds of fewer secondary screenings 
per day, thereby reducing CBPs workload and delays to commerce. The 
actual reduction in the volume of innocent alarms would vary and would 
be greatest at the nation’s busiest ports of entry, such as lx>s Angeles/1 .ong 
Beach, where CBP officials report that PYTs generate up to about 600 
innocent alarms per day. 11 A DNDO official said the requirement for an 80 
percent reduction in innocent alarms was developed in conjunction with 
CBP and was based on a level that would provide meaningful workload 
relief. 

• The primary screening criteria requiring that ASI’s provide at least the 
same level of sensitivity to plutonium and medical and industrial isotopes, 
but not specifying an improvement, were based on DNDO’s assessment 
that PYTs adequately detect such materials, which have a stronger 
radiation signature than HEU.” In addit ion, CBP officials said that 
including medical and industrial isotopes in the criteria addressed a CBP 
requirement for verifying that those transporting certain quantities of 
these materials into the United States are pro|>erly licensed.” 

• The secondary screening requirement that ASPs reduce the probability of 
misidentifying special nuclear material by one-half acklresses the inability 
of relatively small handheld devices to consistently locate and identify 


“About 15 percent of all sea containers arriving in the United States come through Los 
Angelea'Long Beach. In fiscal year 2006, CBP cleared more than 6 million containers 
Uirough the port. 

‘The criteria require an improvement when the radiation emitted by naturally occurring 
radioactive material is used to mask smuggled special nuclear material. Including both 
HEU and plutonium. 

“Far additional information regarding the requirement to verify 1 die legitimacy of 
radioactive material shipments, see GAO. JVuotar Security: NRC and DHS Seed to Tala? 
Additional Stc/ts to Better Track and Detect Radioactive Materials, GAO-08-608 
(Washington, D.C.: June 19, 2008). 
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potent ial threats in large cargo containers. For example, a handheld device 
may fail to correctly ident ify special nuclear material if the material is 
well-shielded or the device is not placed close enough to a radiation 
source to obtain a recognizable measurement According to CBP and 
DNDO, the requirement for a reduction in the average time to conduct 
secondary screenings is not more specific because the time varies 
significantly among ports of entry and types of cargo being screened. 


I)I1S I ncreasec 1 1 lip Improvements to the 2008 round of testing addressed concerns we raised 

about earlier rounds of ASP testing. However, the testing still had 
Rigor Of Advanced limitations, and die preliminary results are mixed. 

Portal Monitor Testing 


Improvements to Testing As we testified in September 2008, DHS’s improvements to the 2008 round 
Provided Credibility to °f ASP testing addressed concerns we raised about previous tests. A 

Test Results particular area of improvement was in the performance testing at the 

Nevada Test Site, where DNDO compared the capability of ASP and 
current -generation equipment to detect and identify nuclear and 
radiological materials, including those that could be used in a nuclear 
weapon. The improvements addressed concerns we previously raised 
about the potential for bias and provided credibility to the results within 
the limited range of scenarios tested by DNDO. For example, we reported 
in 2007 that DNDO had allowed ASP contractors to adjust their systems 
after preliminary runs using the same radiological materials that would be 
used in the formal tests. In contrast, the plan for the 2008 performance test 
stipulated that there would be no system contractor involvement in test 
execution, and no ASP contractors were at the test location on the day we 
observed performance testing. Furthermore, DNDO officials told us. and 
we observed, that they did not conduct preliminary runs with threat 
objects used in the formal tests. In 2007, we reported that DNDO did not 
objectively lest lire handheld identification devices because it did not 
adhere to GBP's standard operating procedure for using the devices to 
conduct a secondary inspection, which is fundamental to the equipment’s 
performance in the field. DNDO addressed this limitation in the 2008 
round of performance testing: CBP officers o|w*rated the devices and 
adhered as closely to the standard operating procedure as test conditions 
allowed. While the test conditions did not allow CBP officers to obtain 
real-time technical support in interpreting the device’s measurements, as 
they would in the field to increase the probability of correctly identifying a 
radiation source, DNDO officials sakl they addressed this limitation. For 
example, they treated a decision by a CBP officer to indicate the need for 
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technical support as a correct outcome if the test scenario involved the 
use of a potential threat, such as HEU. 

Other aspects of testing, while not specifically addressing concerns we 
previously raised, aiso added credibility to the test results. Based on our 
analysis of the performance test plan, we concluded that the test design 
was sufficient to identify statistically significant differences between the 
new technology and current-generation systems when there were 
relatively large differences in performance. Specifically, DNDO conducted 
a sufficient number of runs of each scenario used in the 2008 performance 
testing to identify such differences. 

With regard to the general conduct of the 2008 round of testing, two 
aspects, in particular, enhanced the overall rigor of the tests: (1) criteria 
for ensuring that ASPs met the requirements for each phase before 
advancing to the next, and (2) the participation of CBP .and the DUS 
Science and Technology Directorate. 14 The test and evaluation master plan 
established criteria requiring that the ASPs have no critical or severe 
issues rendering them completely unusable or impairing their function 
before starting or completing any test phase. In addition, the criteria 
established a cumulative limit of 10 issues requiring a work-around (e.g., a 
st raightforward corrective step, such as a minor change in standard 
operat ing procedures) and 15 cosmetic issues not affecting proper 
fiinctioning. DNDO and CBP adhered to the criteria even though doing so 
resulted in integration testing conducted at the Pacific Northwest National 
Laboratory taking longer than anticipated and delaying the start of field 
validation. For example, DNDO and CBP did not allow a vendor’s ASP 
system to complete integration testing until all critical or severe issues had 
been resolved. 

The involvement of CBP and the DHS Science and Technology Directorate 
provided an independent check, within DHS, of DNDO’s efforts to develop 
and test the new (xulal monitors. For example, the lead CBP official 
involved in ASP testing told us that DNDO provided an initial assessment 
of the severity of issues uncovered during testing, but CBP made t he final 
decision on categorizing them as critical, severe, work-around, or 
cosmetic issues. CBP also added a final requirement to integration testing 


'In the case of ASP testing, the Science and Technology Directorate serves as the 
Independent operational test authority, which reports directly to the DHS Under Secretary 
for Management. 
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before proceeding to field validation to demonstrate ASPs’ ability to 
operate for 10 hours without addit ional problems. According to CBP 
officials, their efforts to resolve Issues prior to field validat ion reflect the 
importance CBP places on ensuring that ASPs are sufficiently stable and 
technically mature to operate effectively in a working port of entry and 
thereby provide for a productive field validation. 

The DI IS Science and Technology Directorate, which is responsible for 
developing and implementing the department's test and evaluation policies 
and standards, will have the lead role in the final phase of ASP testing; the 
final phase, consisting of 21 days of continuous operation, is scheduled to 
begin at one seaport alter the completion of field validation. The Science 
and Technology Directorate identified two critical questions to be 
addressed through operational testing: (1) Will the ASP system improve 
operational effectiveness (i.e., detection mid identification of threats) 
relative to the current-generation system, and (2) is the ASP system 
suitable for use in the operational environment at land and sea ports of 
entry? The suitability of ASPs includes factors such as reliability, 
maintainability, mid supportability. Because the operational testing 
conducted at one seaport is not sufficient to hilly answer these 
questions — for example, because the testing will not allow threat objects 
to be inserted into cargo containers— the directorate plans to also conduct 
mi independent analysis of the results from previous test phases, including 
performance testing. 

The 2008 testing still had limitations, which do not det ract from the test 
results’ credibility but do require that results be appropriately qualified. 
Limitations included the following: 

• The number of handheld identification device measurements collected 
during performance testing was sufficient to distinguish wily part icularly 
large differences from ASPs' identification ability. In particular, the 
standard operating procedure for conducting secondary inspections using 
ASPs, which requires less time than when using handheld devices, allowed 
DNDO to collect more than t wice as many ASP measurements mid to test 
ASPs' identification ability against more radiation sources than used to 
test handheld identification devices. 

• The performance test results cannot be generalized beyond the limited set 
of scenarios tested. For example, DNDO used a variety of masking mid 
shielding scenarios designed to include cases where both systems had 100 
percent detect ion, cases where both had zero percent detection, and 
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several configurations in between so as to estimate the point where 
detection capability ceased. 1 However, the scenarios did not represent the 
full range of possibilities for concealing smuggled nuclear or radiological 
material. For example. DNDO only tested shielding and masking scenarios 
separately, to differentiate between the impacts of shielding and masking 
on the probabilities of detection and identification. As a result, the 
performance test results cannot show how well each system would detect 
and identify nuclear or radiological material that is both shielded and 
masked, which might be expected in an actual smuggling incident. 
Similarly, DNDO used a limited number of threat objects to test ASPs’ 
detection and identification performance, such as weaj>ons-grade 
plutonium but not reactor-grade plutonium, which has a different isotopic- 
composition. A report on special testing of ASPs conducted by Sandia 
Nat ional laboratories in 2007 recommended that future tests use 
plutonium sources having alternative isotopic compositions. Sandia based 
its recommendations on results showing that the performance of ASP 
syst ems varied depending on the isotopic composition of plutonium. 

The Science and Technology Directorate’s operational testing is designed 
to demonstrate that the average time between equipment failures (the 
measure of ASPs’ reliability) is not less than 1,000 hours. Thus, the testing 
will not show how reliable the equipment will be over a longer term. DHS 
Science and Technology Directorate officials recognize this limitation and 
said they designed o|)eralional testing only to demonstrate compliance 
with the ASP performance specificat ion. Furthermore, to the extent that 
the Science and Technology Directorate relies on performance test results 
to evaluate ASPs’ ability to detect and identify threats, its analysis of ASPs' 
effectiveness will be subject to the same limitations as the original testing 
and analysis conducted by DNDO. 


Preliminary Test Results The preliminary results presented to us by DNDO are mixed, particularly 
Are Mixed in the capability of ASPs used for primary screening to detect certain 

shielded nuclear materials. However, we did not obtain DNDO's final 
report on performance testing conducted at the Nevada Test Site until 
early April 2009, and thus we had limited opportunity to evaluate the 
report, hi addition, we are not commenting on the degree to which the 
final report provides a fair representation of ASPs’ performance. 
Preliminary results from performance testing show that the new portal 


“Masking is tin* use of naturally occulting radioactive material to make the radiation 
emitted by smuggled material appear to be caused by innocent cargo. In contrast, sliieldirvg 
blocks radiation from being emitted. 
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monitors detected certain nuclear materials better than PVTs when 
shielding approximated DOE threat guidance, which is based on ligtit 
shielding. In contrast, differences in system performance were less notable 
when shielding was slightly increased or decreased: Both the PVTs and 
ASPs were frequently able to detect certain nuclear materials when 
shielding was below threat guidance, and both systems had difficulty 
detecting such materials when shielding was somewhat greater than threat 
guidance. DNDO did not test ASPs or PVTs against moderate or greater 
shielding because such scenarios are beyond both systems' ability. (See 
fig. 3 for a summary of performance lest results for detection of certain 
nuclear materials.) 


Figure 3 Preliminary Results from 2008 Performance Testing for Detection of 

Certain Nuclear Materials 
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With regard to secondary screening. ASPs performed better than handheld 
devices in identification of threats when masked by naturally occurring 
radioactive material. However, differences in the ability to identify certain 
shielded nuclear materials depended on the level of shielding, with 
increasing levels ap|»caring to reduce any ASP advantages over the 
handheld identification devices— another indication of the fundamental 
limitation of passive radiation detection. 

Other phases of testing, particularly integration testing, uncovered multiple 
problems meeting requirements for successfully integrating the new 
technology into operations at |K>rts of entry. Of Uie two ASP vendors 
participating in the 2008 round of testing, one has fallen several months 
behind in testing due to the severity of tlie problem* it encountered during 
integration testing; the problems were so severe that it may have to te<lo 
previous tost phases to be considered for certification. Tire other vendor's 


P**e is 


UAO-09-AS5 Comlmliax Nwlnar SinBKxUnx 


VerDate Oct 09 2002 


09:1 9 Mar 1 7, 201 0 Jkt 000000 


PO 00000 Frm 00076 Fmt 6601 


Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 


PsN: COMSCI 




71 


system completed integration testing, but CBP suspended field validation of 
the system after 2 weeks because of serious performance problems that may 
require software revisions. In particular, CBP found that the performance 
problen*' resulted in an overall increase in the number of referrals for 
secondary screening compared to the existing equipment. According to CBP, 
this problem will require significant corrective actions before testing can 
resume; such corrective actions could in turn change the ability of the ASP 
system to detect threats. Tire problem identified during field validation was in 
addition to ones identified during integration testing, which required multiple 
work-arounds and cosmetic changes before proceeding to the next test phase. 
For example, one problem requiring a work-around related to the amount of 
time it takes for the ASP to sound an alarm when a potential threat material 
lias been detected. Specifications require that ASI*s alarm within two seconds 
of a vehicle exiting the ASP. However, during testing, lire vendor's ASP took 
longer to alarm when a particular isotope was detected Tin* work-around to 
be implemented during field validation requires that all vehicles be detained 
until cleared by the ASF*; the effect on commerce must ultimately be 
ascertained during field vulklatRm. 

CBP officials anticipate that they will continue to uncover problems 
during the first few years of use if the new technology is deployed in the 
field The officials do not necessarily regard such problems to be a sign 
that testing was not rigorous but rather a result of the complexity and 
newness of the technology and equipment. 


Schedule Delays Have 
Allowed More Time 
for Analysis and 
Review of Test 
Results, but DNDO’s 
Latest Schedule Does 
Not Include Computer 
Simulations to 
Provide Additional 
Insight into ASP 
Capabilities 


Ik* lays to the schedule for the 2008 round of testing have allowed more 
time for analysis and review of results, particularly from performance 
testing conducted at the Nevada Test Site. The original schedule, which 
underestimated the lime needed for testing, anticipated completion of 
testing in mid-September 2008 and the DHS Secretary’s decision on ASP 
certification between September and November 2008. D1IS officials 
acknowledged that scheduling a certification decision shortly after 
completion of testing would leave limited time to complete final test 
reports and said the DHS Secretary could rely instead on preliminaiy 
rei>orts if the results were favorable to ASPs. DHS’s most recent schedule 
anticipated a decision on ASP certification as early as May 2000, but DHS 
has not u|xlaled its schedule for testing and certification since suspending 
field validation in February 2009 due to ASP performance problems. 

Problems uncovered during testing of ASPs’ readiness to be integrated 
into operations at U.S. ports of entry have caused the greatest delays to 
date and have allowed more time for DNDO to analyze and review the 
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results of performance testing. Integration testing was originally scheduled 
to conclude in late July 2008 for both ASP vendors. The one ASP system 
that successfully passed integration testing did not complete the test until 
late November 2008— approximately I months behind schedule. (The 
delays to integration testing were due in large part to the adherence of 
DNDO and CBP to the criteria discussed earlier for ensuring that ASPs met 
the requirements for each test phase.) hi cont rast , delays to performance 
testing, which was scheduled to run concurrently with integration testing, 
were relatively minor. Both ASP systems completed performance testing 
in August 2008, about a month later than DNDO originally planned. 

The schedule delays have allowed more time to conduct injection 
S t u di e s — computer simulations for testing the response of ASPs and PVTs 
to the radiation signatures of threat objects randomly “injected" 
(combined) into portal monitor records of actual cargo containers 
transported into the United States, including some containers with 
innocent sources of radiation. However, DNDO does not plan to complete 
the studies prior to the Secretary of Homeland Security’s decision on 
certification even though DNDO and other officials have indicated that the 
studies could provide additional insight into the capabilities and 
limitat ions of advanced portal monitors. According to DNDO officials, 
injection studies address the inability of performance testing conducted at 
the Nevada Test Site to replicate the wide variety of cargo coming into the 
United Slates and the inability to bring special nuclear material anti other 
threat objects to ports of ent ry and place them in cargo during field 
validation. Similarly, while they acknowledged that injection studies have 
limitations, DOE national laboratory officials sakl the studies can increase 
the stat istical confidence in comparisons of ASPs’ and PVTs’ probability of 
detecting threats concealed in cargo because of the possibility of 
sup|K>rling larger sample sizes than feasible with actual test ing. A 
February 2008 DHS independent review team report on ASP testing also 
highlighted the benefits of injection studies, including the ability to 
explore ASP performance against a large number of threat scenarios at a 
pract ical cost and schedule and to permit an estimate of the minimum 
detectable amount for various threats/' 


l *DHS Homeland Security Institute, Independent Review of the Department of Homeland 
Security Domestic Nuclear Detection Office Advanced Spectroscopic Portal: Fi nal Report 
(Feb. 20, 2008). 
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DNDO has the data needed to conduct the studies. It has supported efforts 
to collect data on the radiation signatures for a variety of threat objects, 
including special nuclear materials, as recorded by both ASP and PVT 
systems. It has also collected about 7.000 usable “ st reanvof-commerce’ 
records from ASP and PVT systems installed at a seaport. Furthermore, 
DNDO had earlier indicated that injection studies could provide 
information comparing the performance of the two systems as part of the 
certification process for both primary and secondary screening. However, 
addressing deficiencies in the streanvof-commerce data delayed the 
studies, and DNDO subsequently decided that performance testing would 
provide sufficient information to support a decision on ASP certification. 
DNDO officials said they would instead use injection studies to support 
effective deployment of the new |x>rtal monitors. 


Conclusions Given that radiation detection equipment is already being used at ports of 

entry to screen for smuggled nuclear or radiological materials, the 
decision whether to replace existing equipment requires that the benefits 
of tire new portal monitors be weighed against the costs. DNDO 
acknowledges that ASPs are significantly more expensive than PVTs to 
deploy and maintain, and based on preliminary results from the 2008 
testing, it is not yet clear that the $2 billion cost of DNDO’s deployment 
plan is just ified. Even if ASPs are able to reduce the volume of innocent 
cargo referred for secondary screening, they are not expected to detect 
certain nuclear materials that are surrounded by a realistic level of 
shielding better than PVTs could. Preliminary results of DNDO’s 
performance testing show that ASPs outperformed the PVTs in detection 
of such materials during runs with light shielding, but ASPs’ performance 
rapidly deteriorated once shielding was slightly Increased. Furthermore, 
DNDO and DOE officials acknowledged that the performance of both 
|K)rtaJ monitors in detecting such materials with a moderate amount of 
shielding would be similarly poor. This was one of the reasons that 
performance testing did not include runs with a moderate level of 
shielding. 

Two additional aspects of the 2008 round of testing call into question 
whether ASPs’ ability to provide a marginal improvement in detection of 
nuclear materials and reduce innocent alarms warrants the cost of the new 
technology. First, tire DUS criteria fora significant increase in operational 
effectiveness do not take into account recent efforts to improve the 
current-generation portal monitors' sensitivity to nuclear materials 
through the “energy windowing" technique, most likely at a much lower 
cost. Data on developing this technique were collected during the 2008 
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round of performance testing but have not been analyzed. Second, while 
DNDO made improvements to the 2008 round of ASP testing that provided 
credibility to the test results, its test schedule does not allow for 
completion of injection studies prior to certification even though the 
studies could provide additional insight into the performance of the new 
technology. Without results from injection studies, the Secretary of 
Homeland Security would have to make a decision on certification based 
on a limited number of test scenarios conducted at the Nevada Test Site. 

Assuming that the Secretary of Homeland Security certifies ASPs. CBP 
officials anticipate that they will discover problems with the equipment 
when they start using it in the field. Integration testing uncovered a 
number of such problems, which delayed testing and resulted in ASP 
vendors making multiple changes to their systems. Correcting such 
problems in the field could prove to be more costly and time consuming 
than correcting problems uncovered through testing, particularly if DNDO 
proceeds directly from certification to fall-scale deployment, as allowed 
under the congressional certification requirement that ASPs provide a 
significant increase in operational effectiveness. 


Rpfnmmpndfltinns for recommend that the Secretary of Homeland Security direct the 
CtU CHUdUUi is iui Dlrector of DNDO to take follow ing two actions to ensure a sound 

Executive Action basis for a decision on ASP certification: 

• Assess whether ASPs meet the criteria for a significant increase in 
operational effectiveness based on a valid comparison with PVTs’ full 
performance potential, including the |>otential to farther develop P\Ts' 
use of energy windowing to provide greater sensitivity to threats. Such a 
comparison could also be factored into an u|xlatod cost-benefit analysis to 
determine whether it would be more cost-effective to continue to use 
P\Ts or deploy ASPs for primary screening at particular ports of entry. 

• Revise the schedule for ASP testing and certification to allow sufficient 
time for review mid analysis of results from the final phases of testing and 
completion of all tests, including injection studies. 

If ASPs are certified, we further recommend that the Secretary of 
Homeland Security direct the Director of DNDO to develop an initial 
deployment plan that allows CBP to uncover .and resolve any additional 
problems not identified through testing before proceeding to fall-scale 
deployment— for example, by initially deploying ASPs at a limited number 
of ports of entry. 
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AC6nCV Comments We provided a draft of this report to DOE and DHS for their review and 
J 4 comment. DOE provided technical comments, which we have 

and Our Evaluation incorporated into our report as appropriate. DIIS’s written comments are 
reproduced in appendix II. 

DUS agreed in part with our recommendations. Specifically, DHS stated 
that it believes its plan to deploy ASPs in phases, starting at a small 
number of low-impact locations, is in accordance with our 
recommendation to develop an initial deployment plan that allows 
problems to be uncovered and resolved prior to full-scale deployment. We 
agree that this deployment plan would address our recommendation and 
note that DHS’s comments are the first indication provided to us of the 
depart ment’s intention to pursue such a plan. 

In contrast, DHS did not concur with our recommendations to (1) assess 
whether ASPs meet the criteria for a significant increase in operational 
effectiveness based on a comparison with PVTs’ ftill potential, including 
ftirther developing PVTs’ use of energy windowing, and (2) revise the ASP 
testing and certification schedule to allow sufficient time for completion of 
all tests, including injection studies. With regard to energy windowing. 

DHS stated that using current PVT performance as a baseline for 
comparison is a valid ajiproach because the majority of increased PVT 
performance through energy windowing has already been achieved. While 
DHS may be correct, its assessment is based on expert judgment rather 
than the results of testing and analysis being considered by the department 
to optimize the use of energy windowing. Given the marginal increase in 
sensitivity required of ASPs, we stand by our recommendation to assess 
ASPs against PVTs* ftill |K>tential. DHS can then factor PVTs’ ftill |M>tcntial 
into a cost-benefit analysis prior to acquiring ASPs. On this point, DHS 
commented that its current cost-benefit analysis is a reasonable basis to 
guide programmatic decisions. However, upon receiving DBS's comments, 
we contacted DNDO to obtain a copy of its cost-benefit analysis and were 
told the analysis is not yet complete. 

With regard to injection studies, DHS agreed that the schedule for ASP 
cert ification must allow sufficient time for review and analysis of test 
results but stated that DHS and DOE experts concluded injection studies 
were not required for certification. DHS instead stated that the series of 
ASP test campaigns would provide a technically defensible basis for 
assessing the new technology against the certification criteria. However, 
DHS did not rebut the reasons we cited for conducting injection studies 
prior to certification, including test delays that have allowed more time to 
conduct the studies and the ability to explore ASP performance against a 
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largo number of throat scenarios at a practical cost and schedule. On the 
contrary. DHS acknowledged the delays to testing and the usefulness of 
injection studies. Given that each phase of testing has revealed new 
information about the ca|>abilities and limitations of ASPs, we believe 
conducting injection studies prior to cert ifical ion would likely offer similar 
insights and would therefore l>e prudent prior to a certification decision. 

DHS provided additional comments regarding our assessment of the 
relative sensitivity of ASPs and PVTs and our characterization of the 
severity of the ASPs’ software problems uncovered during field validation. 
With regard to sensitivity. DHS implied that our characterization of the 
relative ability of ASPs and PVTs is inaccurate and misleading because we 
did not provide a con^lete analysis of test results. We disagree. First, in 
meetings to discuss the preliminary results of performance testing 
conducted at the Nevada Test Site. DNDO officials agreed with our 
understanding of the ability of ASPs and PVTs deployed for primary 
screening to detect shielded nuclear materials. Furthermore, contrary to 
tire assertion that a complete analysis requires a eon*>arison of ASPs to 
handheld identification devices, our presentation is consistent with EUlS’s 
primary screening criterion for detection of shielded nuclear materials, 
which only requires that ASPs be compared with PVTs. Finally, while we 
agree that the performance test results require a more complete analysis. 
DNDO did not provide us with its final |>erformance test re|K>rt unt il early 
April 200ft. after DHS provided its comments on our draft report. In the 
absence of the final report, which DNDO officials told us took longer than 
anticipated to complete, we summarized the preliminary results that 
DNDO presented to us during the course of our review as well as to 
congressional stakeholders. 

With regard to ASP software problems uncovered during field validation, 
we clarified our report in res[>onse to DHS’s comment that the severity of 
the problems has not yet been determiner!. DHS stated that its preliminary 
analysis indicates the problems should be resolved by routine adjustments 
to threshold settings rather than presumably more significant software 
■revisions.” However, given the history of lengthy delays during ASP 
testing, we believe that DHS’s assessment of the severity of problems 
encountered during field validation may be overly optimistic. 


As agreed with your offices, unless you (Miblicly announce the contents of 
this report earlier, we plan no further distribution until 30 days front the 
report date. At that t ime, we will send copies to the Secretaries of 
Homeland Security and Energy; the Administrator of NNSA; and interested 
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congressional committees. The report will also be available at no charge 
on the GAO Web site at http://www.gao.gov. 

If you or your staffs have any questions about this report, please contact 
me at (202) 512-3841 or aloisee<**gao.gov. Contact points for our Offices of 
Congressional Relations and Public Affairs may be found on the last page 
of this report. GAO staff who made key contributions to this report are 
listed in appendix III. 



Gene Ak»ise 

Director, Natural Resources 
and Environment 
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The Honorable Joseph I. Lie her man 
Chairman 

Committee on Homeland Security 
and Governmental Affairs 
United States Senate 

The Honorable Henry A. Waxman 
Chairman 

The Honorable John D. Dingell 
Chair Emeritus 
The Honorable Joe Barton 
Ranking Member 

Committee on Energy and Commerce 
House of Representatives 

The Honorable Bennie G. Thompson 
Chairman 

The Honorable Peter T. King 
(tanking Member 
Committee on Homeland Security 
House of Representatives 

The Honorable Edolphus Towns 
Chairman 

Committee on Oversight and Government Reform 
House of Representatives 

The Honorable Bart Gordon 
Chairman 

Committee on Science and Technology 
House of Representat ives 

The Honorable Bart Stupak 
Chairman 

The Honorable Greg Walden 
Ranking Member 

Subcommittee on Oversight mid Investigations 
Committee on Energy and Commerce 
House of Representatives 
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The Honorable Yvette D. Clarke 
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The Honorable Daniel E. Lungren 
Ibinking Member 

Subcommittee on Emerging Threats, 

Cybersecurity, and Science and Technology 
Committee on Homeland Security 
House of Representatives 

The Honorable Charles E. Schumer 
United States Senate 

The Honorable James R. Langevin 
House of Representatives 

The Honorable Michael T. McCaul 
House of Representatives 
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Appendix I: Scope and Methodology 


To evaluate the degree to which Department of Homeland Security’s 
(DHS) criteria for a significant increase in o|>erational effectiveness 
acklress the limit ations of the current generation of radiation detection 
equipment, we clarified the intent of the criteria through Uie Domest ic 
Nuclear Detection Office's (DNDO) written answers to our questions and 
through interviews with U.S. Customs and Border Protection (CBP) 
officials. We also took steps to gain a ftiller understanding of the strengths 
and limitations of the current-generation equipment, which the criteria use 
as a baseline for evaluating the effectiveness of advanced spectroscopic 
portals (ASP). In particular, we obtained copies of the Department of 
Energy (DOE) threat guidance anti related documents used to set 
polyvinyl toluene (PVT) thresholds for detection of nuclear materials. We 
interviewed DOE and national laboratory officials responsible for the 
threat guidance about the process for developing it and the basis for its 
underlying assumptions, including shielding levels. We also interviewed 
DNDO and Pacific Northwest National Laboratory officials regarding the 
extent to which PVTs currently deployed at ports of entry meet the 
guidance and the development and use of energy windowing to enhance 
PVTs’ sensitivity to nuclear materials. 

To evaluate the rigor of the 2008 round of testing as a basis for 
determining ASPs' operational effectiveness, we reviewed the test and 
evaluation master plan and plans for individual phases of testing, including 
system qualification testing conducted at vendors’ facilities, performance 
testing conducted at tlie Nevada Test Site for evaluating ASP detection and 
identification capabilities, and integration testing conducted at Pacific 
Northwest National Laboratory for evaluating the readiness of ASPs to be 
used in an operational environment at |>orts of entry. We also reviewed 
draft plans for fiekl validation conducted at CBP ports of entry and the 
DHS Science and Technology Directorate’s Independent operational test 
and evaluation. In reviewing these documents, we specifically evaluated 
the extent to which the performance test design was sufficient to identify 
statistically significant differences between the ASP and current- 
generation systems and whether DHS had addressed our concerns about 
previous rounds of ASP testing. We interviewed DNDO. CBP. and other 
DHS officials responsible for conducting and monitoring tests, and we 
observed, for one day each, performance testing at the Nevada Test Site 
and integration testing at DOE’s Pacific Northwest National Laboratory. 
We also interviewed representatives of entities that sup|>orted testing, 
including DOE’s National Nuclear Security Administration and Pacific 
Northwest National l^aboratory. the National Institute of Standards and 
Technology, and the Johns Hopkins University Applied Physics 
Laboratory. We review'ed the DHS independent review team report of 
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Appradix I: Sc opr awl MflbodoloQ 


previous ASP test ing conducted in 2007. and we interviewed the chair of 
the review team to clarify the report’s findings. Finally, we examined 
preliminary or final results for the phases of test ing completed during our 
review, and we interviewed DNDO and CBP officials regarding the results. 

To evaluate the test schedule, we analyzed the initial working schedule 
DNDO provided to us in May 2008 :uid the schedule presented in the 
August 2008 test anti evaluation master plan, and we tracked changes to 
the schedule and the reasons for any delays. We interviewed DNIK) and 
other officials with a role in testing to determine the amount of time 
allowed for analysis and review of results. We interviewed DNDO and 
Pacific Northwest National Laboratory officials regarding the injection 
studies, including reasons for delays in the studies and plans for including 
the results as part of the ASP certification process. 

We conducted this performance audit from May 2008 to May 2009 in 
accordance with generally accepted government auditing standards. Those 
standards require that we plan and |>erform the audit to obtain sufficient, 
appropriate evidence to provide a reasonable basis for our findings and 
conclusions based on our audit objectives. We believe that the evidence 
obtained provides a reasonable basis for our findings and conclusions 
based on our audit objectives. 
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Appendix II: Comments from the Department 
of Homeland Security 
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Appendix II: ('oiuiih'iUm from Ike Department 
of llnitn-laiul Security 


*nd PVT systems Tleitfixt, die current cost-benefit analysis is a reasonable basis to guide 
programmatic decisions I lower, given :Ku PVT system* will continue lo be operated hi the 
field, DNDO has initiated efforts to continuously improve the capability of the PVT systems 
These efforts include on ongoing analysis of potential optimization of the PVT energy 
windowing capability ch» will proceed in parallel with other activities relnted to Ceitiftcatson 

Report Recommendation: ’ Revise the schedule for ASP testing and certification to alia* 
sufficient tunc for resti-n and analysts of results from the final phase! of testing ami 
completion of all tests including infection studies ~ 

DIIS docs not concur with aspects of this recommendation 

We agree that there must be sufficient time to review the anal vies and results The current 
schedule for certification is predicated on completing all of the requisite testing and analysts 
of results required to satisfy the criteria for demonstrating significantly improved operational 
effectiveness DIIS has demonstrated that it is implementing a deliberate process to ensure all 
the pre-established exit criteria have been achieved poor to concluding the relevant phase of 
testing. 

However, a team of subyoct matter experts from the Deportment of Energy (DOE) and DHS 
concluded that injection studies were not required for Certification The basis of this decision 
was a rev iew of the senes of test campaigns planned for ASP and the joint determination that 
these tests would provide a technically defensible basis for assessing the ASP technology 
against the Certification criteria "Accordingly . Certification will be based on the whole body 
of experimental knowledge collected on ASP. inc luding technical performance at Nevada 
Test Site (NTS), integration testing at )31-C. field validation al ports of entry , reanalysis of 
selected raw data files through a validated replay tool, and indeprmdcnl Operational Test A 
Evaluation Ncvertheletr, DHS believes that injection studies are useful Mid will continue to 
support and perform them as an aid for deployment decisions and foe future development 


Regarding the conclusions section in your report, wc offer the following additional comments: 

• Conclusions regarding the relative effectiveness of PV7 and ASP to detect Highly 
Enriched Uranium (IIEU) require a more complete analysis than given in your report 
For example, the effectiveness of a system should include the entire system (primary 
plus secondary inspection). Such an analysis would necessarily incorporate a 
comparison of ASP to a hand held RIID Without including the whole system, 
conclusions about overall detection effectiveness are inaccurate and misleading 

• Wc agree with CUP that we will inevitably discos et problems with this highly 
advanced and complex system as it transitions into field operation, there is simply no 
way to Hally anticipate and replicate all real-world peobleins before deployment to the 
field: computer modeling and simulation docs not provide the same level of 
experience or exposure. In antic ipution of tlus fact, deployment will be accomplished 
in phases, starring with a small number of low impact locations As the issues with (lac 
system arc corrected, we will gradually build up lo wider deployment The process 
will be similar to what was used in the successful deployment of PVTs Therefore. 
DHS believes ASP deployment will be implemented in accordance with your 
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Appendix II: Commcnl* from Ihc Department 
of Homeland Swurlly 
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(301077) 


I’M*- 2H 


UAO-OS 655 ( omtuliM Wlrar SiiiBttflln* 


VerDate Oct 09 2002 


09:1 9 Mar 1 7, 201 0 Jkt 000000 


PO 00000 Frm 00091 


Fmt 6601 


Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 


PsN: COMSCI 



86 


PREPUBLICATION COPY 


Evaluating Testing, Costs, and Benefits of 
Advanced Spectroscopic Portals for Screening Cargo at Ports of 

Entry 

INTERIM REPORT 

(ABBREVIATED VERSION) 


Committee on Advanced Spectroscopic Portals 

Nuclear and Radiation Studies Board 
Division on Earth and Life Studies 


NATIONAL RESEARCH COUNCIL 

OF THE NATIONAL ACADEMIES 


THE NATIONAL ACADEMIES PRESS 
Washington, D.C. 
www.nap.edu 


Copynght © National Academy of Sciences All rights reserved 


09:1 9 Mar 1 7, 201 0 Jkt 000000 


Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 


VerDate Oct 09 2002 


PO 00000 Frm 00092 


Fmt 6601 


PsN: COMSCI 




87 


THE NATIONAL ACADEMIES PRESS 500 Fifth Street, N.YV. Washington, DC 20001 

NOTICE: The project that is the subject of this report was approved by the Governing Board of the 
National Research Council, whose members arc drawn from the councils of the National Academy of 
Sciences, the National Academy of Engineering, and the Institute of Medicine. The members of the 
committee responsible for the report were chosen for their special competences and with regard for 
appropriate balance. 

TTiis study was supported by Contract Grant No. HSHQDC-08-A-00056 between the National Academy 
of Sciences and the U.S. Department of Homeland Security. Any opinions, findings, conclusions, or 
recommendations expressed in this publication are those of the aulhor(s) and do not necessarily reflect the 
views of the organizations or agencies that provided support for the project. 

Additional copies of this report are available from the National Academics Press, 500 Fifth Street, N.W., 
Lockbox 285, Washington, DC 20055; (800) 624-6242 or (202) 334-3313 (in the Washington 
metropolitan area); Internet, http://www.nap.edu 

Copyright 2009 by the National Academy of Sciences. All rights reserved. 

Printed in the United States of America 


ii 

Prcpublication Copy 


Copyright © National Academy of Sciences. All rights reserved 


VerDate Oct 09 2002 


09:1 9 Mar 1 7, 201 0 Jkt 000000 


PO 00000 Frm 00093 Fmt 6601 


Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 


PsN: COMSCI 



88 


THE NATIONAL ACADEMIES 

Admen to the Notion on Science, Engineering, and Medicine 

The National Academy of Sciences is a private, nonprofit, self-perpetuating society of distinguished 
scholars engaged in scientific and engineering research, dedicated to the furtherance of science and 
technology and to their use for the general welfare. Upon the authority of the charter granted to it by the 
Congress in 1863. the Academy has a mandate that requires it to advise the federal government on 
scientific and technical matters. Dr. Ralph J. Cicerone is president of the National Academy of Sciences. 

The National Academy of Kngineering was established in 1964. under the charier of the National 
Academy of Sciences, as a parallel organization of outstanding engineers. It is autonomous in its 
administration and in the selection of its members, sharing with the National Academy of Sciences the 
responsibility for advising the federal government. The National Academy of Engineering also sponsors 
engineering programs aimed at meeting national needs, encourages education and research, and 
recognizes the superior achievements of engineers. Dr. Charles M. Vest is president of the National 
Academy of Engineering. 

11k Institute of Medicine was established in 1970 by the National Academy of Sciences to secure the 
services of eminent members of appropriate professions in the examination of policy matters pertaining to 
the health of the public. The Institute acts under the responsibility given to the National Academy of 
Sciences by its congressional charter to be an adviser to the federal government and. upon its own 
initiative, to identify issues of medical care, research, and education. Dr. Ilarvev V. Finebcrg is president 
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Preface 


The threat of a nuclear attack on the United States has haunted the U.S. public 
consciousness and been a central motivation in U.S. national defense since the 1950s. This was 
vividly demonstrated by the image of .American schoolchildren doing “duck and cover” drills at 
the early heights of tensions between the United States and the Soviet Union. With the end of the 
Cold War. the prospect of a full-scale nuclear exchange between superpowers diminished, but 
the specter of new and different threats emerged: nuclear terrorism and clandestine nuclear 
attacks. Countering these new threats is a different kind of challenge and a goal that all 
reasonable people support. The question however, is where to devote limited funds to achieve the 
greatest impact against these risks. This report is an interim report of a study on the testing of 
next generation radiation detectors for screening cargo at ports of entry to the United States, one 
layer of the defense against such attacks. These new detectors are called advanced spectroscopic 
portals (ASPs), 

U.S. Customs and Border Protection (CBP) is responsible for screening cargo for nuclear 
and radiological material at ports of entry. The Domestic Nuclear Detection Office (DNDO) is 
responsible for development and testing of new detectors and coordinating efforts for this 
mission. Both CBP and DNDO are in the Department of Homeland Security (DHS). DNDO 
issued the contract for this study to the National Academy of Sciences (NAS) in late April 2008 
at the direction of Congress. The study is to advise the Secretary of Homeland Security on 
testing, analysis, costs, and benefits of the systems. DNDO wanted the NAS to issue a report in 
just over 4 months, and the NAS was prepared and equipped to deliver a report on that schedule, 
provided that all of the necessary information was provided by DHS. To carry out the study, the 
National Research Council (which is the operating ami of the NAS) assembled a committee with 
expertise in detection and identification of radioactive materials (nuclear materials and devices), 
cost-benefit analysis, statistical interpretation of data, algorithms for analysis of measurements, 
radiation shielding, deployment of detection systems, and port-of-entry operations. 

To gather information for the study, the committee observed operations during visits to 
ports of entry and test sites, reviewed the test plans and results, and met with experts and 
program managers. The committee obviously could not observe the prior tests, and so in addition 
to looking at the test plans and results from those tests, the committee took as valuable input 
reports by the Government Accountability Office (GAO) and the Independent Review Team, 
which was convened at the request of the DHS Secretary. Like the prior tests, those reports were 
completed before the committee was formed, and indeed led to the request for this study. Given 
that DNDO acknowledged several of the problems with earlier testing, the committee focused 
more of its efforts on testing conducted in 2008 and the analysis that followed. The committee 
met in May. June. July, August and October 2008 for information gathering, and subgroups of 
the committee visited ports in Seattle. Los Angeles, and Long Beach; border crossings in Blaine, 
Washington, and Otay Mesa, California; and met with experts at Pacific Northwest National 
Laboratory. Los Alamos National Laboratory, and Sandia National Laboratories in Albuquerque. 
The committee also heard from ASP program staff, the vendors, and outside experts in meetings 
in Washington. D.C. 

The original plan for testing, evaluation, and consultation was a tightly coupled schedule 
dictated by the Secretary of Homeland Security’s intent to make a decision in September 2008 
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PREFACE 


whether to certify that the ASPs would provide “a significant increase in operational 
effectiveness”. 'Ill is wording was in the Department of Homeland Security Appropriations Act 
for Fiscal Year 2007. and certification was required by Congress before DHS obligates funds for 
full-scale procurement of ASPs. This requirement was repeated in fiscal year 2008. In late July 
of 2008. DHS issued a signed memorandum defining what is a significant increase in operational 
effectiveness, in the context of ASP testing. At the same time, it became clear that the equipment 
vendors. DNDO. and CBP could not meet their September target date because testing would not 
be completed until much later. Also, DNDO had not finalized some of the methods for analyzing 
results, and particularly for assessing costs and benefits. In the fall, as testing and evaluation 
continued to take longer than DHS hoped, the NAS proposed to DHS that the committee issue an 
interim report that would help DNDO and CBP complete their testing and evaluation more 
effectively. DBS accepted this proposal. 

At the time that this report entered peer review, the committee had only seen preliminary 
results and analyses from the performance testing and an incomplete version of the DNDO cost- 
benefit analysis, both in briefing form. Because of the preliminary nature of the results the 
committee has seen and the incomplete state of the cost-benefit analysis methodology, this 
interim report focuses more on methodology than on results. During the peer review, DNDO 
provided a draft final report on performance testing. Unfortunately, the DNDO report was 
received too late to be considered in this Academy review. DNDO and DHS still have analysis 
and decisions ahead of them, even alter the analysis of performance testing is finalized, and this 
interim report should help with that work. The final report will address the balance of the 
committee's statement of task. The committee wrote this interim report to assist DHS in its 
procurement efforts, to provide the Secretary with initial advice, and to begin to fulfill Congress* 
request. It is the committee's hope that DNDO, CBP, and DHS will consider the report in the 
spirit it is intended. 

This report is an abbreviated version of the classified report provided to DNDO, DHS, 
and Congress. Some sensitive details have been removed, but the findings and recommendations 
remain unchanged from the full report. 

Robert Dynes, Chairman 

Committee on Advanced Spectroscopic Portals 
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Executive Summary 

To improve screening of containerized cargo for nuclear and radiological material that 
might be entering the United States, the Department of Homeland Security (DHS) is seeking to 
deploy new radiation detectors, called advanced spectroscopic portals (ASPs). The ASPs are 
intended to replace some or all of the current system of radiation portal monitors (called PVT 
RPMs) used in conjunction with handheld radioisotope identifiers (RIIDs) to detect and identify 
radioactive material in cargo. The U.S. Congress required the Secretary of Homeland Security to 
certify that ASPs will provide a ’‘significant increase in operational effectiveness" over continued 
use of the existing screening devices before DHS can proceed with full-scale procurement of 
ASPs for deployment. Congress also directed DHS to request this National Research Council 
study to advise the Secretary of Homeland Security about testing, analysis, costs, and benefits of 
the ASPs prior to the certification decision. The objectives of this study are to: (1) evaluate flic 
adequacy of the past testing and analyses of the ASP systems: (2) evaluate the scientific rigor 
and robustness of the testing and analysis approach: and (3) evaluate the cost-benefit analysis of 
ASP technology. Each of these is discussed below. This interim report is based on testing done 
before 2008: on plans for. observations of. and preliminary results from tests done in 2008: and 
on the agency’s draft cost-benefit analysis as of October 2008. The report provides advice on 
how DHS’ Domestic Nuclear Detection Office (DNDO) can complete and make more rigorous 
its ASP evaluation for the Secretary and the nation. 

Testing: The committee finds that past testing had serious flaws. DNDO has 
acknowledged and addressed a number of those flaws in later testing. The 2008 performance 
tests were an improvement over previous tests: DNDO physically tested some of the limits of the 
systems, although shortcomings remain. DHS needs to address these shortcomings for a rigorous 
approach. 

Scientific Rigor: To make the testing and evaluation more scientifically rigorous, the 
committee recommends an iterative approach with modeling and physical testing complementing 
each other. DNDO’s current approach is to physically test small portions of the threat space 
(possible threat and cargo configurations) and to use other experimental data to test algorithms in 
the systems. However, the set of combinations of threats and cargo configurations is so large and 
multidimensional that DNDO needs an analytical basis for understanding the capabilities of its 
detector systems. In a more rigorous approach, scientists and engineers would use models of 
threat objects, radiation transport, and detector response to simulate performance and use 
physical experiments to validate the models' fidelity and enable developers to refine the models 
iteratively. Much of the foundation for modeling sources, radiation transport, and detector 
response is already in place in the national laboratories. This kind of interaction between 
computer models and physical tests is standard for the development of high-technology 
equipment and is essential for building scientific confidence. 

The idea of an iterative approach also extends to deployment: the committee recommends 
a process for incremental deployment and continuous improvement, with experience leading to 
refinements in both technologies and operations over time. .As a first step in this process DHS 
should deploy its currently unused low-rate initial production ASPs for primary and secondary 
inspection at various sites to assess their capabilities in multiple environments w itliout investing 
in a much larger acquisition at the outset. 
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2 EVALUATING TESTING, COSTS. & BENEFITS OF ASPs: INTERIM REPORT 

Cost-Benefit: DHS* definition of a “significant increase in operational effectiveness” is a 
modest set of goals. Preliminary estimates indicate that the cost increases from replacing the 
PVT/RIID combination with ASPs outweigh the cost reductions from operational efficiencies. 
Therefore, a careful cost-benefit analysis will need to reveal the advantages of ASPs among 
alternatives. The cost-benefit analysis was not complete when this report was written, but it 
should include three key elements: a clear statement of the objectives of the program: an 
assessment of meaningful alternatives: and a comprehensive, credible and transparent analysis of 
in-scope benefits and costs. The committee recommends that DHS not proceed with further 
procurement until it has addressed the findings and recommendations in this report and the ASP 
is shown to be a favored option in the cost-benefit analysis. 
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Summary 

Containerized cargo entering the United States at sea ports and land-border crossings for 
trucks is currently screened for radiation using detectors, called radiation portal monitors (RPMs) 
made from a plastic scintillator, called PVT, 1 in conjunction with handheld radioisotope 
identifiers (RIIDs). The Department of Homeland Security (DHS) is seeking to deploy new 
radiation detectors, called advanced spectroscopic portals (ASPs), to replace the PVT and RIID 
combination, which has known deficiencies. Title IV of division E of the Consolidated 
Appropriations Act. 2008 (Public Law 1 10-161) requires the Secretary of Homeland Security to 
submit to Congress a report certifying that a “significant increase in operational effectiveness*’ 
over continued use of the existing screening devices will be achieved with the ASP before “funds 
appropriated under this heading shall be obligated for full-scale procurement of Advanced 
Spectroscopic Portal Monitors.” DHS is testing and evaluating the ASPs to inform the 
Secretary’s certification decision. If the Secretary certifies the ASPs, DHS may purchase more 
than one billion dollars worth of ASPs. The net lifecycle cost of these ASPs could be more than 
twice that figure. 

Hie U.S. Congress directed DHS to request that the National Research Council of the 
National Academies conduct a study prior to certification to: (1) evaluate the adequacy of the 
past testing and analyses of the ASP systems performed by DHS’s Domestic Nuclear Detection 
Office (DNDO); (2) evaluate the scientific rigor and robustness of DNDO's current testing and 
analysis approach: and (3) evaluate DNDO's cost-benefit analysis of ASP technology. Due to 
delays in the test and evaluation program, the Academies and DHS agreed that the study 
committee would issue an interim report that provides the committee's evaluation of testing 
plans and execution it has seen, and advice on how DNDO can complete and make more 
rigorous its ASP evaluation for the Secretary and the nation. 

'Iliis interim report is based on testing done before 2008, plans lor and preliminary results 
from tests done in 2008, and the agency's draft cost-benefit analysis as of October 2008. The 
committee received briefings on the performance test results and analysis and on the cost-benefit 
analysis, but the committee did not receive written reports on those topics by February 2009, 
when the interim report entered the Academy peer review process. The committee addresses 
each element of the study task below. 

PAST PERFORMANCE TESTING 

Performance tests prior to 2008 had serious flaws that were identified by the Government 
Accountability Office and the Secretary’s ASP Independent Review Team. All truck-conveyed 
containers at ports and border crossings pass through a PVT portal which constitutes primary 
screening, and those trucks that trigger an alarm are sent to secondary screening, which is 
conducted with a PVT portal and RIID. The tests prior to 2008 did not adequately assess the 
capabilities of the ASP systems in primary and secondary screening compared with the currently 
deployed PVT and RIID screening systems, nor whether the ASP systems met criteria for 
procurement. DNDO utilized the same sources in performance testing that were used to set up 
and calibrate this testing. The number of sources available was small, but this is not sufficient 


1 PVT stands for polyvinyl toluene. 
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4 EVALUATING TESTING. COSTS. & BENEFITS OF ASPs: INTERIM REPORT 

reason to use the same sources for both set up and testing. Device setup and any calibration must 
use separate sources from those used for testing. A component of the standard operating 
procedures for the RIIDs in secondary screening was not followed in the performance tests, 
which disadvantaged the RIID in comparisons with ASPs. 

2008 PERFORMANCE TESTING 

In describing and discussing the tests with the committee. DNDO staff acknowledged 
several pre-2008 deficiencies. According to the 2008 test plan and briefings to the committee in 
Washington. D.C., and at the Nevada Test Site, these deficiencies were corrected. This is 
consistent with the committee's observations of tests and questioning of test personnel. 

Because they have large detectors and because of their configuration. ASPs would be 
expected to improve isotope identification, and provide greater consistency in screening each 
container, greater coverage of each container, and increased speed of screening over that of the 
PVT/RIID combination when used in secondary screening. Consequently, tests of ASPs in 
secondary screening focused on confirming and quantifying that advantage for several threat 
objects, cargos, and configurations. 

When used for primary screening, an ASP system must be compared to the existing 
combined primary and secondary screening system (both PVT and RIID) because of differences 
in standard operating procedures for primary screening (ASPs in primary have an identification 
function). DNDO's preliminary analysis did account for this difference. 

The 2008 performance tests were an improvement over previous tests. DNDO physically 
tested some of the limits of the systems. However, the following shortcomings remain. (1) 
Without modeling to complement the physical experiments, the selected test configurations are 
too limited: (2) the sample sizes are small and limit the confidence that can be placed in 
comparisons among the results: and (3) in its analysis, some of the performance metrics are not 
the correct ones for comparing operational performance of screening systems. These 
shortcomings are described in greater detail w ithin the report. For these reasons. DHS cannot 
conclude definitively whether ASPs will consistently outperform the current PYT-R1ID systems 
in routine practice until the shortcomings are addressed. Better measurement and characterization 
are a necessary first step but may not be sufficient to enable DHS to conclude that the ASPs meet 
the criteria DHS has defined for achieving a “significant increase in operational effectiveness.” 
Hie committee recommends modifications to the current DHS approach to the evaluation 
procedure. These modifications would influence subsequent procurement steps. 

RECOMMENDED APPROACH FOR TESTING AND EVALUATION 

To make the testing and evaluation more scientifically rigorous, the committee 
recommends an iterative approach with modeling and physical testing complementing each 
other. The threat space— that is. the set of possible threat objects, configurations, surrounding 
cargoes, and conditions of transport — is so large and multidimensional that DNDO needs an 
analytical basis for understanding the capabilities of detectors for screening cargo. DNDO’s 
current approach is to physically test small portions of the threat space and to use other 
experimental data to interpolate and extrapolate throughout the threat space to test the 
identification algorithms in the detector systems. 
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For a more rigorous approach. DNDO should use theory and models of threat objects, 
radiation transport, and detector response to simulate performance and predict outcomes. Then 
DNDO can use physical experiments to validate the predictions and allow a critique of the 
models* fidelity to reality. This would enable developers to refine the models iteratively. With 
validated models. DNDO can evaluate the performance of the ASP systems over a larger, more 
meaningful range of cases and threat space than is feasible with physical tests alone. 

This kind of interaction between computer models and physical tests is standard for the 
development of some high-technology equipment and is essential for building scientific 
confidence. The performance tests conducted in 2008. and even prior to 2008. can be used to 
help refine and validate models. 

RECOMMENDED APPROACH FOR THE PROCUREMENT PROCESS 

Hie idea of an iterative approach extends to deployment too. Hie committee noted that 
DHS's testing philosophy is oriented toward a one-time certification decision in the near future. 
However, the mandate for passive radiation screening of cargo at ports of entry is expected to 
continue indefinitely. Rather than focusing on the single decision about the deployment of ASPs, 
the current testing should be viewed as a first step in a continuous process of improvement and 
adaptation of the systems. The threat environment, the composition of container cargo, 
technological and analytical capabilities, and the nature of commerce at the ports of entry have 
changed significantly over the last decade and can be expected to evolve in both predictable and 
unpredictable ways in the coming years. DHS should develop a process for incremental 
deployment and continuous improvement, with experience leading to refinements in both 
technologies and operations over time, rather than a single product purchase to replace current 
screening technology. The process should be developed to address and exploit changes. This 
would result in a system that can be adapted and updated continuously so that it would not be 
outdated by the time all of the ASPs are deployed. 

As the first step in this process DHS should deploy its currently unused low-rate initial 
production ASPs for primary and secondary inspection at various sites as extended operational 
testing. Such deployment, even on this limited scale, would provide additional data concerning 
their operation, reliability, and performance, and allow- DHS to better assess their capabilities in 
multiple environments without investing in a much larger acquisition at the outset. 

Hie development of the hardware for radiation detection and the software for analyzing 
the signals from the detectors is separable. It has been useful to have a competitive approach for 
the combined systems and to see the results. However, as DUS moves forward, it should match 
the best hardware to the best software (particularly the algorithms), drawing on tools developed 
for the competition and elsewhere, such as the national laboratories. 

ASPs will not eliminate the need for handheld detectors with spectroscopic capabilities. 
Because some of the improvement in isotope identification offered by the ASPs over the RIIDs 
is a result of software improvements, the best softw are package also should be incorporated into 
improved handheld detectors. Newer RIIDs with better software might significantly improve 
their performance and expand the range of deployment options available to CBP for cargo 
screening. 

By separating these elements and engaging the broader science and engineering 
community, DHS would have increased confidence in its procurement of the best product 
available w ith current technology, and simultaneously could advance the state of the art. 
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EVALUATING TESTING, COSTS, & BENEFITS OF ASPs: INTERIM REPORT 


RECOMMENDED APPROACH FOR COST-BENEFIT ANALYSIS 

The preliminary analysis presented to the committee suggests that benefits of deploying 
the ASPs may not be clearly and undeniably greater than the costs. Because DNDO’s 
preliminary estimates indicate that the cost increases from replacing the PVT RIID combination 
with ASPs outweigh the cost reductions from operational efficiencies, it is important to consider 
carefully the conditions under which the benefits of deploying ASPs justify the program costs. A 
cost-benefit analysis (CBA) can provide a structure for evaluating whether a proposed program 
(such as the ASP program) is reasonable and justified. 

The Secretary's decision on ASP certification is to rely, at least in part, on whether the 
ASPs meet the objectives in DHS* definition of “significant increase in operational 
effectiveness” (SIOE); however, other factors relating to the costs and benefits of the proposed 
ASP program w ill also need to be taken into account. DHS* definition of a SIOE is a modest set 
of goals: As noted above, the increases in operational efficiency do not by themselves appear to 
outweigh the cost increases from replacing the PVT RIID combination with ASPs, based on 
DNDO's preliminary estimates, and the criteria do not require significantly improved ability to 
detect SNM in primary screening (see Sidebar 3.1). If the .ASPs meet the defined criteria and are 
able to detect the minimum quantities of nuclear threat material that DOE recommends (the 
“DOE guidance”), DHS still will not know whether the benefits of the ASPs outweigh the 
additional costs associated with them, or whether the funds are more effectively spent on other 
elements of the Global .Architecture. 

A CBA can provide insight about the effects of alternative decisions, whether the benefits 
of a given program exceed its costs, and which choices are most cost-effective. To do this, the 
cost-benefit analysis needs to include three key elements: (1) a clear statement of the objectives 
of the screening program; (2) an assessment of meaningful alternatives to deploying ASPs; and 
(3) a comprehensive, credible and transparent analysis of in-scope benefits and costs. 

The CBA should begin by stating clearly what operational problem the .ASPs are 
intended to address. Tliis statement will define the role that the system plays in providing a layer 
in the defense against the importation of a nuclear or radiological device. It should include a 
narrative that clarifies how r the task of improving detection for containers at ports of entry to the 
United States fits into a larger effort to implement or improve detection capabilities, in 
recognition of the many ways that materials could be brought into the United States through 
ports of entry that are not already screened, or across uncontrolled stretches of border. 
Furthermore, to be useful in a procurement decision, a CBA will need to address whether funds 
are better spent to replace the currently deployed equipment rather than to expand coverage to 
other pathways that currently have no radiation screening. This is needed in the ASP CBA 
because it is not evident that it has been done elsewhere. 

The CBA needs to account for meaningful alternatives (including non- ASP programs) to 
reveal the scale of the benefits of ASPs for radiation screening and determine whether these 
benefits outweigh the additional costs. The complexity of the container screening task provides 
opportunities for many different options worthy of consideration. These options include 
variations on deployment configuration and operational processes, and application of 
technologies beyond the PVT RIID and ASP detectors such as improved versions of existing 
handheld passive detectors (deploying handhelds with state-of-the-art software) and advanced 
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methods for detecting nuclear materials. Considerations should include active interrogation, 
improved imaging systems, and integration of existing technologies. 

These alternatives need to be compared to a baseline that reflects as realistically as 
possible the screening capability that DHS currently has in place. Thus, the baseline should 
reflect the number and placement of PVT and RIID detectors, sensitivity of the sensors based on 
how they are operated at each port, and performance of existing handheld detectors in the 
manner they are used in the field. Such an analysis would indicate what capability an investment 
in ASPs will provide beyond the existing systems as they are currently deployed and operated or 
beyond alternative technologies that could be developed and deployed for radiation detection. 

In comparing these alternatives, it is important that the cost-benefit analysis treat benefits 
and costs in a comprehensive, credible, and transparent manner. The benefit assessment should 
show how this program contributes to improving security with respect to prevention of the 
detonation of a nuclear device or radiological weapon in the United States. Because this is the 
primary objective of the ASP program, a cost-benefit analysis that is silent on this subject would 
be incomplete. Such an assessment is difficult and no assessment of such benefits will be 
definitive or unassailable, however it remains important to consider these factors. The cost 
assessment should cover all phases of the acquisition life cycle in a manner that is independent of 
contractor or program office biases and assess the risk of cost escalation associated with the 
estimate. 

The committee recommends that DHS not proceed with further procurement until it has 
addressed the findings and recommendations in this report and the ASP is shown to be a favored 
option in the cost-benefit analysis. 
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Introduction 

In 2007. more than 11 million cargo containers arrived on ships and were offloaded at 
U.S. sea ports. An approximately equal number arrived by truck and another 2.75 million arrived 
by rail across land borders. The previous year, the SAFE Port Act (P.L. 109-347) was signed into 
law and required that “not later than December 31, 2007, all containers entering the United 
States through the 22 ports through which the greatest volume of containers enter the United 
States by vessel shall be scanned for radiation. To the extent practicable, the Secretary shall 
deploy next generation radiation detection technology.” Cargo screening at ports of entry to the 
United States 2 is carried out by U.S. Customs and Border Protection (CBP) in the Department of 
Homeland Security (DHS). 'Hie Domestic Nuclear Detection Office (DNDO, also in DHS) 
coordinates federal, state, and local detection efforts to address the threat of nuclear terrorism, 
and develops, procures, and supports the deployment of detection equipment within the United 
States. One of DNDO’s chief clients is CBP. This report concerns efforts to develop, test, and 
deploy next generation radiation detection technology. The following paragraphs provide some 
history of events that preceded the request for this study. 

DNDO requested proposals for the next generation of radiation detectors for cargo 
screening (called advanced spectroscopic portals, or ASPs) from commercial vendors. DNDO 
selected three vendors for full testing, awarding contracts worth up to SI. 2 billion for both 
testing and acquisition. The Government Accountability Office (GAO) and others raised 
questions about the reliability of DNDO’s testing of the devices. Consequently, Congress 
restricted use of the funds for “full-scale procurement of Advanced Spectroscopic Portal 
Monitors” until the Secretary of Homeland Security submits to Congress “a report certifying that 
a significant increase in operational effectiveness will be achieved” by deploying ASPs to 
replace the screening devices that are already in place. 3 

The GAO has on-going audits of the .ASP testing and procurement program and has 
raised several objections to the way the program, including its testing, evaluation, and life-cycle 
cost analyses have been conducted (GAO 2006; 2007a, 2008a). as well as criticisms of the larger 
“global architecture” of which the cargo screening is a piece (GAO 2008b; 2009). In August 
2007. the DHS Secretary formed a group to carry out an independent review. Iliat group issued 
its draft final report in November 2007. 4 In December 2007, the 2008 Consolidated 
Appropriations Act (P.L. 110-161) stated “(t)hat the Secretary of Homeland Security shall 
consult with the National Academy of Sciences before making such certification.” In its Joint 
Explanatory Statement accompanying the legislation, Congress clarified its intent and this 
statement was the basis for the committee's statement of task (see Appendix A). 

Hie ASP testing and evaluation program encountered some delays in 2008. which 
delayed any NAS report but created an opportunity for the NAS committee to provide input on 


1 “A Port of Entry is any designated place at which a CBP officer is authorized to accept entries of merchandise to 
collect duties, and to enforce the various provisions of the customs and navigauon laws (19 CFR 1011) ” 

3 Sec Title IV of division E of the Consolidated Appropriations Act, 2008, Public Law 110-161 

4 The Independent Review Team's final report was issued in February 2008 Some of its findings are discussed in 
Chapter 3. 
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how testing and evaluation and the cost-benefit analysis should be completed. This interim report 
provides that advice to support future decisions by the Secretary' of Homeland Security 
concerning development, certification, and deployment of ASPs. This chapter describes the 
origin of the study, the broader context of the threat of nuclear terrorism, and the currently 
deployed system for screening cargo containers for radiation. Chapter 2 gives readers who are 
not familiar with technologies for radiation detection some background on how detectors work. 
Chapter 3 provides the committee's views on ASP testing and analysis conducted by DHS 
offices both prior to 2008 and during 2008, including findings and recommendations on how to 
complete the work. Chapter 4 provides the committee's findings and recommendations for 
completing the ASP cost-benefit analysis. A final report will contain the committee's findings 
and recommendations on DNDO's completed tests and analyses. 

WHY SCREEN FOR RADIATION? TIIE THREAT OF NUCLEAR TERRORISM 

The possibility of nuclear terrorism has become more credible as it has become clearer 
that non-state actors may have or be able to acquire the means for a nuclear attack: gaining the 
knowledge of how to design a weapon, the materials for a nuclear explosive, and the ability to 
deliver and detonate the device. .After the attacks on the United States on September 1 1. 2001. 
there is little doubt that well-funded, well-organized, and capable groups have the motive and 
intent to carry out high-consequence attacks on the United States. The knowledge of how to 
build a nuclear explosive is increasingly seen as a small hurdle, as designs of simple weapons 
have been discovered in non-nuclear weapons states, and given reports that A.Q. Khan's black 
market nuclear distribution network offered a weapon design, in addition to designs and 
equipment for uranium enrichment. 5 Production of special nuclear material (SNM)- the fuel for 
a nuclear explosive is still generally thought to require the resources of a nation, but the 
material could be acquired by other means, such as theft or black market sales. After the collapse 
of the Soviet Union, the United States and Russia agreed to work together to ensure that 
scientists with weapons-design and production expertise remain in Russia, and not sell their 
expert services to others. They agreed to begin to account for and secure weapons-grade material 
in states of the former Soviet Union and to emplace radiation detectors to catch special nuclear 
material illicitly leaving Russia (the second line of defense). It became evident through this 
cooperation that the Soviet Union had not kept careful records of its inventory of special nuclear 
material at several dozen locations, so it is unknown whether material was already stolen from 
the stockpiles. 6 

To detonate a nuclear device on U.S. soil (including smuggled weapons, improvised 
nuclear devices, or dirty bombs), a terrorist must either acquire the necessary materials within the 
United States or smuggle them across U.S. borders. One potential path would be to bring the 
material in through one of the 327 official ports of entry' into the United States, including land, 
air. and seaports, concealed as apparently ordinary cargo. 

Each day in 2007, U.S. container ports handled an average of 71,000 twenty-foot 
equivalent units (TEUs. a measure of container size) of cargo. 8 In addition, an average of 22.000 


5 See. eg. Corera (2006). 

6 See. for example, reports from the National Research Council on materials protection control and accounting (NAS 
2009, 2007. 2005a, 2005b, 2005c, 1999, and 1997) 

7 In this case, “container ports” refers to sea ports, and excludes cargo coming into the United States via land border 
crossings. 
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truck and rail containers entered the U.S. by land each day in 2007. In fact, an average of one in 
nine containers carrying global trade is bound for or is coming from the United States (USDOT, 
2007). 

According to testimony from Jayson Ahem, acting Commissioner of U.S. Customs and 
Border Protection, in March 2009, radiation portal monitors (RPMs) in place now scan about 
98% of the shipping containers entering U.S. maritime ports. 96° o of trucks at Northern land 
border crossings, and 100% of those at Southern border crossings. 9 Additional monitors are 
being installed in the remaining ports and border crossings, and plans are in development to 
cover rail lines. This is a significant accomplishment. However, it is only a first step. The system 
does not cover small water vessels or general aviation, and much uncertainty remains about how 
to improve the overall capability of the system to reduce the threat posed by nuclear terrorism 10 
in view of ever-increasing technological innovations and limited financial resources. 

EFFORTS TO INTERDICT NUCLEAR MATERIALS A I PORTS OF ENTRY 

The U.S. government —both the administration and Congress — concluded that it would 
be valuable to screen people, luggage, vehicles, and cargo entering the United States for nuclear 
and radiological material. U.S. Customs and Border Protection put in place a system of RPMs 
that use passive devices to detect radioactive material entering the country. Typical RPMs at a 
small border crossing are shown in Figure 1.1. In the towers on each side of the roadway or 
traffic lane are two panels (one high, one low) containing radiation detectors. The RPMs use 
PVT plastic scintillation detectors, which detect gamma rays emitted by most radionuclides, but 
have a limited ability to characterize the source of those gamma rays. The PVT detectors are 
capable of measuring only crude spectral information. The RPMs also have neutron detectors, 
which can detect neutron-emitting materials, such as plutonium. 

Cargo screening is just one of several overlapping layers of defense against unlawful 
import of nuclear material, none of w hich offers perfect protection. The layered defense system 
begins with securing the materials in the facilities where they reside overseas and has additional 
layers for detecting and preventing smuggling efforts at foreign nations' borders and interdicting 
in transit. The Department of Energy, through the Second Line of Defense and other programs, 
uses many of the same detectors as CBP but deploys them overseas at border crossings and sea 


8 The numbers cited for container traffic can be confusing The mantime industry counts twenty-foot equivalent 
units (TEUs) when counting cargo containers of varying lengths — a forty fool container is two TEUs — but others 
count actual containers, or even conveyances In this report. TEUs will only be used to describe overall container 
traffic for sea ports 

0 Statement of Jayson P. Ahem. Acting Commissioner. U.S Customs and Border Protection, Department of 
Homeland Security before the Committee on Appropriations, Subcommittee on Homeland Security, April 1. 2009 
10 In 2008. David Maurer of the Government Accountability Office testified that the Department of Homeland 
Security’s Domestic Nuclear Detection Office (DNDO) “lacks an overarching strategic plan to help guide how it 
will achieve a more comprehensive architecture " (GAO 2008b) 
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(a) (b) 

Figure I I (a) The tall pillars closest to the foreground in this photograph are RPMs at a land 
border crossing between Canada and New York, (b) A truck is shown passing through a series of 
RPMs and ASPs on a test track. SOURCE: CBP (2008). 


ports under agreements with the foreign governments where they are located. 1 1 The final layer is 
at our borders' ports of entry, with the RPMs (and the associated hand-held radiation detectors). 
Over 1070 RPMs were in operation at U.S. ports of entry, as of July 2008. Hand-held 
radioisotope identification devices (RIIDs) ,: also were in operation at ports of entry at that time. 

Every container of foreign origin carried by a truck passes through screening. At sea 
ports, the procedure is not totally consistent at each site or for each container. Containers may be 
loaded onto a chassis which is then connected to a tractor that drives the container through an 
RPM and off the terminal. Containers destined for rail transport may be carried by a truck to a 
nearby location with a rail line w here the train is built (some of these are screened w ith a RPM 
when the truck is pulling them) or they may be loaded directly onto rail cars (so-called on-dock 
rail or roll-on. roll-off rail loading). Mobile detectors are used for some of the containers that are 
not conveyed by truck. The ASP-C RPMs are only used for containers conveyed by truck. 

The current concept of operations (CONOPS) for screening of cargo containers for 
radioactive material consists of a two-stage screening process. In the first stage, primary 
screening, the container is driven through a PVT RPM. When an RPM used in primary screening 
detects radiation levels above a gamma-ray or neutron alann threshold, the container is diverted 
to a lane dedicated to secondary screening. Because there is radioactive material in a small but 
significant fraction of ordinary cargo, radiation alarms in primary' screening are quite common. 
This radioactive material includes naturally occurring radioactive material (NORM), 1 * as well as 

11 The National Nuclear Security Administration, a semi-autonomous agency within the Department of Energy 
(DOE), runs these programs The committee refers to DOE here and throughout tire report for simplicity The 
similarity and overlapping nature of the DOE and CBP-DNDO programs has led DNDO to consult and cooperate 
with DOE on some aspects of tire ASP program 

11 llie term "isotope identification" or "radioisotope identification" is commonly used, although it is usually not 
technically corTcct. It is only meaningful to refer to an isotope in the contest of a specific element The same is true 
of the tenn radioisotope. A nuclide or a radionuclide may be any isotope of any element. In this report, the terms 
"isotope" and "radioisotope" are synonymous with nuclide and radionuclide, respectively, consistent with common 
usage. 

13 NORM comprises many materials derived from rocks, such as granite tabic tops, porcelain, and kitty litter, and 
matenals high in potassium, such as bananas and potassium chloride (salt substitute). 
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radiopharmaceuticals used in medicine and industrial radiation sources Even when radiation is 
not detected at a primary RPM, secondary actions may be taken based on independent 
information about the cargo or a CBP agent’s judgment that the cargo is suspect 

In secondary screening, the container is driven through another RPM and examined with 
a “spectroscopic” detector, which in principle is capable of identifying specific radioactive 
substances The spectrometer currently in use is a handheld radioisotope identification device 
(RED. see Figure 1.2) 

One or more CBP officers examine the container with a RED to identify whether the 
source is NORM, an industrial source, medical radionuclides, a threat object, or some 
combination of these CBP may decide to send the spectrum electronically to a centralized group 
of specialists, called Laboratories and Scientific Services (LSS), for analysis CBP may also open 
the container and visually inspect the contents as well as further monitor the contents with the 
RED At some ports of entry, the container may also be subject to additional inspections such as 
imaging with an X-ray type machine (a radiography device with a gamma or X-ray source) to 
look for localized heavy metal objects (shielding or SNM). and direct examination of the cargo, 
including removal from the truck or shipping container 



Figure 12: A handheld RED SOURCE DNDO (2008a) 

Committee members observed secondary screening operations at two border crossings 
and three ports. The committee’s observations were consistent with descriptions given in 
briefings to the committee by CBP in May and October A truck carrying a container that 
triggers a primary alarm may be delayed by 5 to 15 minutes or more, depending on the 
configuration of the port of entry and the relative ease or difficulty of identifying the source of 
radiation First, because of the layout of the primary and secondary screening areas, at some 
ports of entry it may take several minutes for a truck stopped in primary screening to be diverted 
to secondary screening At some ports of entry, it requires that a CBP officer stop all lanes trafFic 
through the RPMs to allow the truck that caused the alarm to cross to the secondary screening 
area Switching to ASPs would not reduce this delay for a truck that triggers a primary alarm, but 
to be certified for primary screening, ASPs must alarm on fewer trucks The result of deploying 
ASPs that meet the criteria would be some reduction in the time spent in screening overall 
Screening a truck with a handheld RED may take several minutes or more, depending on how 
quickly the alarm can be resolved 

However, the time required to carry out screening is only part of the picture of actual 
operations at ports of entry CBP has stated repeatedly that the current system of radiation 
screening, using PVT RPMs and REDs, does not impede the flow of commerce The 
committee’s observations were, again, consistent with those statements. In no case that the 
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committee observed was there a backup of trucks resulting from radiation screening. Other 
steps manifests and immigration at border crossings, and safety inspections at border crossings 
and ports — had trucks waiting. While an alarm on the primary screening detectors sometimes 
stopped traffic for all of the lanes, typically it resulted in no net delay for the trucks that did not 
trigger the alarm. This is because the queue at the next inspection station usually had not yet 
cleared. DNDO and CBP officials also told the committee that replacing the current system with 
an ASP system w ould not reduce the number of CBP officers who conduct radiation screening at 
ports of entry. 
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2 

Background on Radiation Detection 

Portal monitors contain detectors for both ganinia rays and neutrons. There are thousands 
of known radionuclides, and most of them emit one or more gamma rays, so most radioactive 
materials emit some gamma rays. A single radionuclide can emit one or many distinct gamma 
rays, each having a characteristic energy and intensity, 14 resulting in a gamma ray spectrum 15 
characteristic of that radionuclide. The intensity depends on the probability of emission in each 
decay event and the amount of the nuclide present. 16 The ability to reliably determine the 
presence of a radionuclide, especially in the actual or potential presence of other radionuclides, 
depends on having a detector with sufficient sensitivity and energy resolution. 

The neutron detectors employed in the radiation portal monitors (RPMs) do not resolve the 
energies of the neutrons. However, this is not a major drawback because relatively few 
radionuclides or combinations of radionuclides emit neutrons, and nearly all of those sources are of 
interest for security reasoas. The detection of neutrons, then, is a strong indicator of the presence of 
threat material and the need to interdict the truck. Although many of these radionuclides and their 
daughters also emit alpha or beta particles, only gamma rays and neutrons are sufficiently 
penetrating to be detectable outside of a shipping container that holds the radiation source. 

When screening cargo, the Department of Homeland Security (DHS) tries to identify 
whether the cargo contains radionuclides useable in a radiological or a nuclear weapon. 1 The 
radionuclides of greatest concern for radiological attacks have been identified in several studies, 
including those by the U.S. Nuclear Regulatory' Commission and the U.S. Department of Energy, 
the International Atomic Energy Agency, and others (see NRC-DOE 2003; I.AEA 2003; NAS 
2008). and include americium-241, cesium- 137, cobalt-60, iridium- 192, and strontium-90. Some 
of these radionuclides are easy to detect if they are present in significant quantities. For example, 
cobalt-60 emits two relatively high energy gamma rays with each disintegration, one at 1173 
keV and one at 1333 keV. Strong cobalt-60, cesium- 137. and iridium- 192 (used in radiography) 
sources require heavy shielding to enable people to work near them. 

The materials of greatest concern for nuclear explosive devices are called direct-use 
nuclear materials — materials that are directly useable in a nuclear explosive device (this includes 
special nuclear material: uranium-233, uranium-235, and plutonium) — and do not necessarily 
require heavy shielding. For example, uranium-235 emits one intense gamma ray of energy 
185.7 keV in 57.2% of its disintegrations. Plutonium-239 emits numerous weak (low-intensity 
or low-probability) gamma rays. The strongest, most readily detectable of these have energies of 


14 For example, iridium-192 emits dozens of gamma rays as it decays. 4 of which are intense (iridium-192 emits them 
in approximately 30 percent or more decays) Iodine- 1 31. used in medicine, emits one intense gamma ray (emitted in 
over 80 percent of decays) and 1 7 other gamma rays (emitted in between 0.00009 and 7 percent of decays) 

15 A gamma ray spectrum, the set of gamma rays of different energies emitted by a source, is represented as a plot of 
the number of gamma rays versus energy 

16 The relative intensities of multiple gamma rays from a single radionuclide can also be used to help identify it 

17 A radiological weapon uses radioactive material to cause harm based on the radiation the material emits. A 
nuclear weapon uses nuclear reactions to release large amounts of energy in a nuclear explosion, which also releases 
radioacUvc material A radiological weapon is unlikely to kill many people, but can cause harm and economic 
damage A nuclear weapon is the most devastating weapon in the U.S arsenal 

14 
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51.6. 98.4. 129.3, 375.1, 451.5, 650 (a “multiplet" containing about a dozen gamma rays of 
nearly the same energy), and 769.3 (doublet) keV. Plutonium also emits neutrons because of 
spontaneous fission. No real material is purely composed of one radionuclide. Other nuclides, 
including other radionuclides in many cases, are present because they are byproducts of the 
creation of the radionuclide or because they are decay products of the main radionuclide. Highly 
enriched uranium (HEU) contains, by definition, at least 20 percent uranium-235, with the rest 
typically being uranium-238 and trace quantities of uranium- 234. Even weapons-grade uranium 
(generally considered to be at least 90 percent uranium-235, and what a weapons state would use 
in a nuclear weapon) may contain up to 10 percent uranium-238. The composition of plutonium 
typically has even more isotopes in nieasureable quantities: some mix of plutonium-238. -239. - 
240. -241. and possibly -242. 

A notional gamma-ray spectrum would show up simply as a curve with peaks (spikes of 
counts) at the characteristic gamma-ray energies, but zero counts everywhere else. As discussed 
below, the w idth of the peaks, or “energy resolution" is different for different types of detectors. 
Real spectra are necessarily more complicated, due to the existence of alternative physical 
mechanisms for absorption and scattering of gamma rays in the detector and. to a lesser extent, to 
imperfections in the way different types of detectors and individual detectors of the same type 
operate. The most important difference between a real spectrum and the above-described 
“notional" spectrum is the presence of a broad continuum of gamma rays caused by Compton 
scattering. 18 For a gamma ray of given energy, the continuum lies below the peak and has a 
predictable shape, based on the gamma-ray energy and the composition and size of the detector. 
The continuum tends to fill in the regions between the peaks, and can make it difficult to identify 
peaks if the peaks are broad (i.e.. in low -resolution detectors), and or the peaks are weak 
compared to the continuum. Additionally, as noted, shielding can attenuate the peaks and add to 
the continuum, and additional radiation from natural background and masking materials can 
introduce additional gamma-ray peaks and add to the “Compton" continuum. The combination 
of these effects complicates the spectrum and creates a formidable challenge to the identification 
of radionuclides, especially w ith detection systems of relatively low resolution like thalium- 
activatcd sodium iodide |NaI(Tl), sodium iodide or Nal for short) detectors. Thus the challenge 
of testing the ability of a system to detect and identify a particular source under varying 
conditions is great. This chapter describes important technical aspects of passive detectors used 
to detect radiation from sources located in cargo containers. 

SHIELDING 

The observed gamma-ray spectrum from a source, (e.g., special nuclear material. SNM) 
is influenced by the presence and distribution of surrounding materials, which attenuate and 
scatter gamma rays by absorption and Compton scattering. These materials can include, 
containers, other materials being shipped with the source or placed around it in an attempt to 
shield it from detection (shielding), and an air gap. Attenuation even occurs in the radioactive 
material itself (self shielding). High-energy gamma rays are attenuated less than low-energy 

18 Compton scattering is a fundamental physical process in which a gamma ray scatters off an electron, giving up 
some of its energy to the electron and retaining the rest in the scattered gamma ray When this occurs in the detector 
(and the scattered gamma ray exits the detector without further interaction), the detector “sees" the energy of the 
scattering electron When it occurs in material outside the detector (e.g . shielding), the scattered gamma-ray might 
be detected by the detector Because the amount of gamma-ray energy given up to the electron varies continuously, 
the result in cither case is a contribution to the continuum in the detector 
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gamma rays, and high-atomic number (high-Z) materials, such as tungsten and lead attenuate 
more than lovv-Z materials, such as aluminum and wood. High density materials also attenuate 
more than low-density materials, and there is a correlation between pure high-Z materials and 
high-density materials, and between pure low-Z materials and low-density materials. Shielding 
can reduce the intensity of gamma-rays observed by the detector (both peaks and continuum), 
and also shift the continuum to lower energies and increase its intensity relative to the peaks. 

Self Shielding 

An important consideration for detection of nuclear materials is self shielding. Because 
uranium and plutonium are very heavy elements with large number of electrons (high-Z). they 
strongly absorb gamma rays. Gamma rays produced in the interior of a thick piece of nuclear 
material are likely to be absorbed within the material. This effect makes it more difficult to 
detect these materials. 

HEU emits very few neutrons, virtually all of them from the small percentage of 
uranium-238 present in the material. 19 Consequently, one cannot reliably detect HEU with a 
neutron detector. Plutonium emits neutrons, most of which are emitted by the isotope plutonium- 
240. Self-shielding has little effect in diminishing neutron emission, because sub-critical 
multiplication actually increases the neutron emission. Testing of RPMs carried out with SNM 
has demonstrated the detection of plutonium with some shielding and. in some tests. HEU. With 
sufficient shielding, passive detectors would fail to detect even large quantities of these 
materials. 


MASKING 


Masking is the phenomenon that occurs when benign radioactive materials obscure the 
signature of a radionuclide of interest. This occurs when the benign radionuclide either 
overwhelms the detector with a stronger signal or creates spectral signals that compromise the 
algorithm’s ability to analyze the spectra. Multiple radionuclide sources in the cargo, including 
masking materials, produce spectra that are linear sums of the spectra of individual 
radionuclides. The geometry of the source and masking materials can affect the spectrum, 
because different radioactive materials can be located in different positions relative to the 
detectors and any shielding materials. 

In addition, the interaction between the radioactive material and the shielding or masking 
material may result in secondary emissions that could confound identification. When testing for 
the effects of shielding, both high-Z and lovv-Z materials should be investigated. High-Z 
materials are more effective at attenuating gamma rays, especially low-energy gamma rays, 
whereas low-Z materials, notably materials containing hydrogen atoms, enhance the absorption 
of neutrons. 2 ' 1 High-Z materials close to a source that emits beta particles, such as strontium-90, 
will enhance the production of Bremsstrahlung. a continuum spectrum of photons (gamma rays) 
resulting from the stopping of electrons. If shielding and masking are used in combination, it is 
important to consider scenarios where the masking material is closer to the source than the 


19 HEU containing very small amounts of chemical impurities emits some neutrons, produced by reactions of alpha 
particles on light elements in these impunties 

10 Neutrons lose more of their kineUc energy in collisions with low-Z nuclei than with high-Z nuclei, and low-energy 
neutrons are much more likely to be absorbed than high-energy neutrons 
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shielding and vice versa, because the order may affect the observ ed gamma ray spectrum and the 
angular dispersion of scattered gamma rays. 

Finally, all radioactive decay data are subject to statistical variations, which are 
particularly significant for weak sources/ 1 

HOW DETECTORS WORK 

Gamma-ray detectors and neutron detectors are used in both the proposed Advanced 
Spectroscopic Portals (ASPs) and the currently-used PVT portals. Both systems use moderated 
helium-3 proportional counters for neutron detection. For gamma-rays, the ASP system uses 
thallium-activated sodium iodide (Nal(TI), sodium iodide or Nal for short] detectors for gamma- 
ray detection, and the PVT system uses polyvinyl toluene based plastic scintillation detectors for 
ganima-rav detection. The plastic scintillator is made of a polyvinyl toluene solvent with a 
(typically) p-terphenyl solute. After mixing the two materials, the solvent is polymerized to 
make the plastic. .Another system proposed for the ASP portals uses high purity germanium 
(IlPGe) detectors for gamma-ray detection. How each detector works is discussed in the 
follow ing paragraphs. 

Sodium Iodide Detectors 

Sodium iodide detectors consist of a Nal crystal containing approximately 0.1% thallium, 
coupled optically to a photomultiplier tube. 22 They are scintillation detectors: gamma rays 
interact with the detector to produce low -energy photons in the energy range of visible light, a 
phenomenon called scintillation. The Nal crystal is transparent to light. A scintillation photon is 
captured by the photomultiplier, which converts it into an electron, which is then accelerated and 
amplified in the photomultiplier to produce many electrons. The total electrical signal at the 
output of the photomultiplier is related to the sum of the photons from across the cry stal and is 
roughly proportional to the energy deposited in the detector by the gamma ray, so the size of this 
signal is logged and tallied as the energy' of one gamma ray. Nal detectors are expensive 
compared to plastic detectors, but inexpensive compared to HPGc detectors. (See below.) 

The range of energies detected for the full-energy gamma-ray peak in a Nal detector is 
typically around 8% of energy FWHM. 23 In other words, the peak in a Nal detector sped mm 
from a 1 MeV gamma ray might be 80 keV wide. This is relatively low (poor) energy resolution 
for gamma spectroscopy. When several different gamma ray's are closer together in energy than 
the detector resolution, as can be the case with some sources observed by Nal detectors, it is 
difficult to identify them all. 'litis is particularly true for a weak gamma ray (one with few counts 
observed) close to a stronger one. Another problem with a low-resolution detector, such as Nal, 
is that a weak gamma ray peak can be difficult to observe above the Compton continuum from 


21 Radioactive decay is measured by detector counts of emitted particles and is modeled most naturally and rather 
faithfully by the Poisson distribution with standard deviation equal to the square root of its mean, hence a good estimate 
of the statistical variation in the number counts N is VN . and the variation relative to the count is VN/N = 1/VN. 

" Originally, single crystals were grown for the detectors. Currently many of the detectors are made of a 
polycrystalline material that has better resistance to cleavage from mechanical or thermal shock. 

J FWHM stands for full width at half maximum, the width (energy spread) of the peak in the spectrum at half the 
height of the peak above any underlying continuum The low resolution of Nal detectors is the result of low 
efficiencies in the conversion of gamma-ray energy to energy of the light photons, and a low yield of electrons 
(around 0. 1 5 per photon) at the photocathode 
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higher energy gamma rays, because the few counts in the peak are spread of a range of energies. 
Figure 2.1 shows a typical gamma-ray spectrum of naturally occurring radioactive material 
(NORM) measured with a Nal detector. Low detector resolution poses a challenge to analysis 
algorithms necessary to process the data and obtain meaningful conclusions, especially when 
there is a large statistical uncertainty in the data (i.e.. few counts). 


NORM (Laboratory Room Background) 



0 200 400 600 000 1000 1200 t400 1600 1900 2000 2200 2400 2600 2900 3000 3200 3400 3600 

Energy (keV) 

Figure 2.1 A gamma-ray spectrum gathered from the background radiation in a laboratory using 
a sodium-iodide detector. The x-axis is the energy in keV and the y-axis is the number of 
gamma-ray detections counted within a particular energy range. Gamma-ray peaks from 
common background radionuclides are labeled on the figure. 
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PVT Detectors 

Like Nal detectors. PVT detectors are scintillation detectors. Unlike crystals, plastic 
scintillators can easily be fabricated into large detectors, and are relatively inexpensive. The 
larger size permits the detection of a larger number of events from the same gamma-ray source. 
However, the low density, low light yield, and especially the low atomic number 24 of the plastic 
scintillator combine to make the detector much less effective than Nal for spectroscopic 
measurements. They provide only crude information about the gamma-ray energy. Figure 2.2 
shows a typical gamma-ray spectrum of radionuclides measured with a PVT detector, illustrating 
the absence of observ able full-energy peaks. 



Figure 2.2. A calibration gamma-ray spectrum gathered by a PVT portal monitor. The 
background has been subtracted. SOURCE: Stromswold et al. (2004). 


HPGe Semiconductor Detectors 

Gamma-ray spectrometers based on high-puritv germanium (HPGe. or germanium) 
detectors are widely used as laboratory scientific instruments. Their energy resolution is typically 
around 0.1-0.2°o FWHM of the gamma-ray energy, nearly two orders of magnitude better 
(narrower peaks) than a Nal detector. An HPGe detector is a semiconductor ionization-type 
detector, which operates on a different principle from Nal and PVT detectors. In an ionization 
detector, the gamma-ray energy is converted directlv into electrons, which form the signal 
proportional to the energy deposited by the gamma ray.* 

Figure 2.3 shows the HPGe spectrum of the same NORM source whose measurement 
with an Nal detector was shown above (Figure 2.1). Hie advantage of the higher-resolution 


24 Materials with low atomic numbers have almost no photoelectric interactions with gamma raj's and therefore 
exhibit no full energy peak 

25 The high resolution of germanium detectors results from efficient conversion of the gamma-ray cncigy into electrons. 

Prepublication Copy 

Copynghl C National Academy of Sciences Ail rights reserved 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00124 Fmt 6601 Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 




20 


EVALUATING TESTING. COSTS. <& BENEFITS OF ASPs: INTERIM REPORT 


NORM 



Figure 2.3 A gamma-ray spectrum gathered with a germanium detector from the background 
radiation in the same laboratory' mentioned in Figure 2.1. Again, the \-axis is the energy in keV 
and the y-axis is the number of gamma-ray detections counted within a particular energy range. 
Gamma-ray peaks from common background radionuclides are labeled on the figure. 


detector is evident. (Note that the presence of uranium-235, whose strongest gamma-ray is 
buried in the continuum with a Nal detector, is clearly revealed in the IIPGe spectrum.) 

Although the higher energy resolution of HPGe detectors is essential in many laboratory 
measurements and would be desirable for detecting nuclear and radiological materials, especially 
under difficult conditions (e.g.. masking), these detectors have other characteristics that make 
their widespread use in RPMs problematic. The main drawbacks are the difficulty of producing 
detectors in very large sizes needed to detect relatively small amounts of radiation in a short 
time, and the high cost per detector, which makes it expensive to use large numbers of them in an 
RPM. Also, the detectors must be cooled to low' temperatures, requiring liquid nitrogen or 
special, mechanical or thermoelectric cooling devices. 

Neutron Detectors 

Neutron detection in the RPMs use helium-3 proportional counters, a type of gas-filled 
ionization detector that has built-in amplification caused by a complex process of charge 
multiplication. The detectors are embedded in polyethylene, which acts as a “moderator.” 
slow ing (“thermalizing”) the neutrons emitted by sources of interest to low energies. 26 


26 In the detector, the neutron reacts with a helium-3 nucleus to produce an energetic proton (p or hydrogen-1 ) and a 
triton (t or hydrogen-3). This reaction has a low cross-section (probability) for all but low-energy neutrons, so 
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Because the neutrons must be slowed down to make them detectable, the counter does 
not measure the energy of the incident neutrons. Hi is means that the detectors measure no useful 
spectroscopic information about the neutrons. Because very few radionuclides emit neutrons, and 
almost all of them are of security interest, this is not a serious drawback. Neutrons are simply 
counted. The mere detection of any neutrons above the low natural neutron background counting 
rate signals a likely cause for concern . 2 

ANALYZING SPECTRA 

Once a gamma-ray spectrum has been collected, it must still lie analyzed to identify the 
radionuclide(s) that generated the radiation. Two different strategies have been employed for this 
analysis in the ASPs: peak matching and template matching. Each has advantages in some 
configurations. It is also likely that the available algorithms could be improved by involving 
more of the science and engineering community to work on these problems. 

As described in the previous section, a detector pulse-height spectrum for a 
monoenergetic gamma-ray source has a peak centered on the full energy of the incident gamma 
rays and a continuous tail at lower energies caused by Compton scattering. A peak-matching 
algorithm identifies the full-energy peak and matches that energy to the signature energies in its 
library of radionuclides. Many radionuclides have multiple characteristic gamma rays, and more 
than one radionuclide may be present in cargo, so the algorithm must be able to identify and 
match multiple peaks in a single spectrum. 

The advantage of peak matching is that there is alw ays a full-energy peak that is separate 
from the Compton tail. One disadvantage is that only a fraction of the detector counts are in the 
full energy peak. Peaks are also obscured by the Compton distribution of higher-energy gamma 
rays. Also, attenuating material between the source and the detector can drastically reduce the 
number of full-energy gamma rays that even reach the detector, making it difficult to 
differentiate full-energy peaks from the background. 

A template-matching algorithm has a library of energies of gamma rays emitted in 
radioactive decay and also a library of full detector spectra from radionuclides with intervening 
attenuating materials. Template matching compares not just the full-energy peak, but the whole 
spectrum to its libraries. .An advantage of template matching is that all of the detector counts are 
used toward identification, and the effect of shielding can be accounted for. at least 
approximately. The challenge in template matching is the nearly limitless set of combinations of 
sources and attenuating materials and thicknesses, along with background radiation. 

Although software implementing algoritlims for gamma-ray spectral analysis has been 
the subject of intense development in the national laboratories, and several vendors of 
spectrometers provide such software, there are in fact few commercial products available for 
radionuclide identification using gamma spectroscopy . 28 A particular problem is the dearth of 


neutrons must be slowed down for them to be detected well The reaction energy is earned off by the proton and 
triton, which lose their energy by ionizing atoms in the detector gas The ionized atoms make an electrical signal that 
is amplified by the proportional counter 

r One likely source of neutrons is plutonium Neutron sources such as the isotope californium-252 and mixtures of 
natural or man-made alpha-emitting isotopes with beryllium are used in some applications, including downhole 
measurements in oil wells 

31 Isotope identification software and algorithms are different from the ASP software for interaction with the 
operational hardware (occupancy sensors, gate arms, etc.) The former are exchangeable modules that analyze 
spectral data found in data files that follow standard formats. The latter arc specific to each vendor’s ASP 
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commercial software for the complex problem of analyzing sources that can be shielded and 
masked with low-resolution (Nal) detectors — a problem of current interest mainly to detection 
systems for nuclear and radiological materials. Even the scientific literature on this topic is 
sptirsc. Engaging the broader science and engineering community in this challenge could lead to 
more sophisticated analytical methods from statistics and signal processing being applied to 
radionuclide identification, resulting in better algorithms. 

Radiation Detectors at Ports of Entry Today 

As noted in Chapter 1. the RPMs currently in use are PVT plastic scintillation detectors. 
Because these detectors are inexpensive and easily fabricated in large volumes, they can be made 
to be quite sensitive to radiation. But PVT detectors have very poor energy resolution: they 
cannot distinguish one gamma ray energy from another, except over broad energy ranges, so they 
have very limited ability to characterize the source of those gamma rays. The PVT detectors in 
the RPMs at most ports of entry have been equipped with crude energy resolution in the form of 
energy windowing: The gamma-ray events are binned into four large energy windows. Although 
these energy w indow s are too broad for isotope identification, the ratios of the counts in different 
windows and to background levels in the same window help to identify the presence of radiation 
sources that require further examination. 29 The RPMs are also equipped with moderated helium- 
3 neutron detectors. The RPMs alarm if the container occupancy causes the RPM to exceed a 
gross gamma-ray counting threshold, exceed an allowed gamma-rav energy windowing ratio 
value, or exceed a gross counting threshold for neutrons. 

'Hie spectrometer currently in use in the secondary inspection is a handheld radioisotope 
identification device (RIID) which contains a small Nal detector. 

At some ports of entry, the container may also be subjected to additional interrogation 
inspections such as imaging with an X-ray type machine (a radiography device w ith a gamma or 
X-ray source) to look for localized heavy metal objects (shielding or SNM). and direct 
examination of the cargo, including removal from the truck or shipping container. This or other 
suspicious results can trigger additional inspections. 

The gamma ray alarm threshold (the count rate above which the alarm is triggered) is 
established for each port based on the threat guidance and a number of other factors. 30 
Performance of the RPMs relative to the threat guidance is tested by measurements using 
standard sources that are not special nuclear material, but have gamma-ray signatures that are 
similar to that of plutonium or uranium, and so can serve as surrogates. CBP has said that the 
threshold is selected to balance the needs for sensitivity for commerce to flow. Although most 
RPM gross- gamma-count thresholds are set to meet a particular guidance level, a fraction of 
them are set to a different level. Using the energy windowing mentioned above, PNNL reports 
that all of the RPMs can detect a plutonium surrogate source that is lower than the guidance 
activity (i.e., the RPM is more sensitive than the plutonium guidance). 

NEXT GENERATION RADIATION DETECTION TECHNOLOGIES: ASPs 


29 All deployed SAIC RPMS PVT systems have energy windowing algorithms that use four energy wtndows CBP 
also has some Ludlum RPMs that only have two windows and hence 1 ratio on which to alarm 

30 The threat guidance, which is classified, was established by the Department of Energy in a 2003 letter to Pamcy 
Albright, assistant secretary of homeland security for science and technology 
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The goal of DIIS in replacing the PVT RPM systems with the ASP technology is to 
address three perceived needs (Test and Evaluation Master Plan. August 2008): 

• “To improve the detection of nuclear weapons and radiological nuclear threat sources 

• To reduce the burden associated with unnecessary inspection of conveyances with only 
naturally-occurring radioactive material (NORM) 

• To improve the consistency and accuracy of the identification of nuclear weapons and 
radiological/nuclear threat sources” 

Specifically, the ASP performance specifications called for systems that can detect and identify 
SNM, weapon-indicating radionuclides. NORM, medical radionuclides, and industrial 
radionuclides alone and in combination. According to DIIS. the portal detection systems should 
respond consistently and predictably, and should assist CBP personnel in determining whether to 
release a conveyance or to detain it per the agency’s standard operating procedure or concept of 
operations (CONOPS. from the Performance Specifications July 2007). The ASP systems are 
expected to detect and identify these threat materials when surrounded by “engineered shielding 
or masking and or significant amounts of cargo." Improved RPMs, such as ASPs, should tolerate 
a wide range of conditions including variations in natural background radiation, environmental 
stress and weather conditions, and should be able to accommodate low- and high-volume traffic 
areas. 

Benign sources of radiation in normal commerce (such as medical radionuclides) would 
not need to be sent to secondary inspection if they could be identified in the primary inspection. 
The ASPs were developed to provide both detection and identification of radiation sources in 
cargo containers. The portals use Nal or IIPGe detectors, which provide greater differentiation 
in the detector response to gamma rays of different energies than PVT. With suitable softw are to 
analyze the gamma-ray spectrum, the source of the gamma rays can, in principle, be identified. 
At the time that the committee prepared this report, the HPGe ASP had not met requirements to 
undergo full testing by CBP and DNDO. 31 so the committee's report focuses on the Nal systems. 

There are reasons to believe that the ASP could perform the functions now being 
performed in both primary and secondary screening, in most cases. A confident identification of 
NORM in primary screening would significantly reduce the number of referrals to secondary 
screening. For cases in which primary screening determines that the cargo is suspect, an ASP 
could be used also in a secondary screening w ith the container moving at a lower speed to obtain 
greater statistical accuracy and hence more effective identification. CBP is also considering a 
hybrid deployment, with some ASPs deployed primarily in high traffic ports, and retaining PVTs 
in other, lower-traffic ports, and using ASPs for secondary screening at all ports. 

The ASP has advantages over the combination of a PVT portal and RIID detector in 
secondary inspections. Hie detection and identification feature of the ASP is enhanced by the 
slower speed in secondary as compared to the primary. The ASP is larger than the RIID and 
therefore can collect comparable or better statistical spectral data (e.g., higher counts and hence 
lower relative statistical variation), and the ASP has better identification software. The ASP 


31 As noted above, gamma-ray energy resolution in a HPGe detector is far superior to that in a Nal detector, but the 
cost of HPGe crystals is much higher than sodium iodide crystals The cost and the difficulty of growing large HPGe 
crystals resulted in the HPGe ASP having a much smaller detector volume than the others A smaller detector 
requires more time being exposed to get a statistically useful number of counts (detection events) Consequently the 
HPGe ASP could not meet the requirement to screen cargo containers passing at the speeds required in the systems 
specifications. Because it could not meet these criteria, the contract with the vendor of the HPGe ASP was not 
extended A change in CONOPs to allow for longer exposure times could enable the HPGe detectors to operate in 
secondary screening, but performance with different CONOPs has not been evaluated in the DNDO program 
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localization of the source is better than that of the RIID because the ASP uses data from a 
continuous screening and can analyze time-slices of data. The larger detector has much better 
coverage of the cargo container. Most containers will spend less time in secondary inspection 
with an ASP because the slow-speed scan w ill confirm the presence of radiation sources that are 
only NORM, so that the manual (hand-held detector) survey would not be necessary unless the 
container is unloaded and a RIID would be used to investigate specific packages in the container. 

The ASPs are required to identify as well as to detect the radioactive material. As a result, 
assessment of the performance of the instrument is not limited to the sensitivity of the detectors, 
but also includes determining the level of confidence in the threat identification algorithm for 
each system. Although there is evidence that the spectral analysis programs work remarkably 
well under challenging circumstances, the two vendors* algorithms appear to yield somewhat 
different results, and it is not clear at this point that either is optimal. 
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Testing and Analyses of the ASP and PVT/RIII) Systems 

The committee was asked to evaluate the adequacy of past testing and analyses of the 
advanced spectroscopic portal (ASP) systems performed by the Department of Homeland 
Security's (DHS’s) Domestic Nuclear Detection Office (DNDO), and the scientific rigor and 
robustness of DNDO’s testing and analysis approach. Hie Joint Explanatory Statement from 
Congress states that the intent of the Secretary of Homeland Security's consultation with the 
National Academies is to “bring robustness and scientific rigor to the procurement process." As 
noted at the beginning of this report, w hen the committee ended its information gathering for this 
interim report in mid-January, the testing and analyses were incomplete and DNDO had not 
provided written reports describing test results. No one on the study committee observed ASP 
tests before the committee was formed in May 2008. This chapter is based on the committee's 
observations in visits to ports of entry and test sites, reports of testing done before 2008 and 
documented plans for 2008 tests, observations of performance tests conducted in 2008 at the 
Nevada Test Site, and a briefing (October 8. 2008) on preliminary results from performance tests 
done in 2008. 

The Government Accountability Office (GAO), DHS's Independent Review Team (IRT), 
and Congress already have reviewed and criticized pre-2008 testing of ASPs and PVT RI IDs. 
The criticism resulted in the requirement for additional testing to support a decision about 
procurement of ASPs. Another factor that led to the requirement for DNDO to revisit testing in 
2008 is that Customs and Border Protection (CBP) was dissatisfied with the ASP systems' 
reliability and compatibility with other CBP systems. Systems qualification testing, and 
particularly systems integration testing, were more rigorous and demanding in 2008. lhese tests 
took much longer than expected and only one vendor had successfully completed systems 
integration testing, as of January 2009. 

DNDO. CBP. and their contractors have conducted many tests over the last three years. A 
list of the major tests conducted on the ASPs and RPMs can be found in Table 3.1. DNDO has a 
complex set of criteria to evaluate. The characterization of a system is a process, and no one set 
of tests is expected to describe thoroughly all variables. Indeed, the scientific method describes a 
cycle of hypothesis and experimentation, w hich when applied to instrument development, allows 
for an iterative process of identification and mitigation of weaknesses. How the tests could be 
better crafted to carry out this process is described in detail later in this chapter. 

Hie process for testing radiation portal monitor systems, such as the ASP systems, begins 
at the component level and progresses to the subsystem and system level. Initial testing is 
conducted with components and subsystems in the laboratory, such as functional and 
environmental testing of individual detector elements, graduating to larger subsystems mid full 
systems in systems qualification testing. The last of these is done at Pacific Northwest 
Laboratory. Overall systems performance is measured with live radiation sources and a 
simulated port of entry at the Nevada Test Site (NTS, see Figure 3.1), mid field validation testing 
is conducted outdoors at U.S. ports of entry with representative container cargo loadings. 
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Tabic 3.1 Tests and Key Questions 


Tests 


Description 


NYCT Tests 


Special 
(“Blind" or 
“Demo") 
Testing 


Phase 3 Tests 


Environmental 

Product 

Qualification 

Testing 


Systems 

Qualification 

Tests 


Performance 
Tests at NTS 


ASP and PVT portals were 
installed in primary' and 
secondary screening sites. 
The data collected were 
used for modeling and 
injection studies. 

Set of 1 2 “relatively blind" 
test configurations. Tests 
performed at NTS. 
Anticipated results were 
compared to results given 
to the operator When 
available, underlying data 
(raw spectra) were 
evaluated by third party 
isotope identification 
algorithm These results 
were compared to operator 
results. Statistical analysis 
was performed by NIST to 
determine how special test 
results compared to 
standard test results 
Tests performed at NTS 
with various sources and 
attenuating materials in 
cargo containers moving at 
different speeds. 

Tests took place at the 
vendor's facility and at a 
National Recognized Test 
Laboratory and witnessed 
by government 
representatives 
A series of tests designed 
by the vendors and 
approved by DNDO to 
assure that the system 
requirements of the 
performance specification 
have been met Tests took 
place at the vendor's 
facility and PNNL's 33 1 G 
facility and were witnessed 
by government 
representatives 
Cargo containers loaded 
with varying configurations 
of shielding material, 
masking material, threat 
objects, and surrogate 
sources arc run on a 
roadway flanked by the 


Objective 

To collect data (spectra) 
on stream of commerce 
cargo containers to feed 
into injection studies 


Kc> Quest ions 

What does radiation in the stream 
of commerce look like? What is the 
range and variation in radiation 
emitted by typical cargo? 


To assess vulnerabilities in 
the performance test plan 
To evaluate the possibility 
that bias had been 
introduced into the test 
results by vendors or the 
test team 

To provide additional data 
to the vendors for system 
development 


Has bias been introduced into the 
ASP test results by cither vendors 
or the test team? 

Docs the test plan contain enough 
of a diversity of sources and test 
configurations? 


To aid in development of How do known areas for 
secondary screening improvement affect the 

operations and procedures performance of ASPs, and what can 
be done to address them? 


Verify that the system can 
function within the 
environment, including 
weather and climate, in 
which the system will be 
operated and maintained 
Verify technical 
achievement of the system 
requirements as described 
in the Performance 
Specification for ASPs 


Are all components of the ASP 
system durable enough to withstand 
the climate and environmental 
stresses at ports of entry (POEs) 
across the country? 

Have the basic system requirements 
been met? Is the system ready to 
enter performance testing? 

Is the ASP system suitable and 
deployable within the existing 
nuclear detection architecture? 


Evaluate system 
performance and collect 
data to support operational 
test and evaluation. 

- Compare ASP system 
performance with that 
of the PVT and RJID 


How do the ASP systems perform 
relative to the current generation of 
detection and identification 
systems? 

W’hat are the thresholds for 
detection of threat materials 
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Integration 
! ests 


Field 

Validation Test 


Operational 

Test 


PVT and ASP detectors in 
sequence Secondary RIID 
screening is carried out in 
the staging area 


Tests conducted by DNDO 
at PNNL's331G test 
facility. Test systems were 
placed in a simulated port 
of entry’ environment and 
evaluated for compatibility 
with CBP standard 
operating procedures 
(SOP) and other 
equipment, such as gate 
arms and traffic lights 
Both hardware and 
software were evaluated 
Test conducted at ports of 
entry Conducted by CBP 
with ASP systems m place 
screening the stream of 
commerce trucks PNNL 
will draft the final report 


ASP systems will be placed 
at a POE in both primary 
and secondary locations in 
conjunction with PVT 
monitors to screen stream 
of commerce cargo 
containers The systems 
will be operated by CPB 
officers using standard 
operating procedures A 
survey of CBP personnel 
will also occur 


systems 

- Characterize the effect 
of shielding and 
masking on ASP and 
RIID performance 
against threat objects 
and NORM 

- Collect data to support 
verification of system 
requirements 

Collect data in support of 
operational testing and 
evaluation requirements 
Demonstrate that the ASP 
systems are ready to be 
integrated into the 
interdiction sy stems at 
U S POEs for field 
validation in primary and 
secondary configurations 


• Perform system 
installation procedures 
and process 

- T rain officers in the use 
of the sy stem 

- Familiarize officers with 
operations of ASP 
systems with PVT 
systems 

- Conduct operations with 
ASP alone 

Validate the operational 
effectiveness and 
suitability of ASP at ports 
of entry under realistic 
operating conditions 


How do the systems perform with 
threat sources in the presence of 
masking and attenuating material? 


Do the ASP systems meet the 
necessary integration requirements 
associated with their deployment, 
and arc they suitable for operator 
use? 


Does the ASP system fit readily 
into the existing POE RPM sites? 
Arc they suitable for operator use? 
Is the ASP system interoperable 
with users/ stakeholders to execute 
the nuclear detection and reporting 
mission? 


How effective is the ASP system in 
terms of lime to conduct screening, 
number of referrals to secondary 
screening, involvement of LSS. and 
reliability, availability, and 
maintenance of the system 11 Have 
CBP personnel identified any 
concerns or limitations of the 
system 

Is the ASP system interoperable 
with uscrs/stakcholdcrs to execute 
the nuclear detection and reporting 
mission? 

Is the ASP system suitable and 
deployable within the existing 
nuclear detection architecture 
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(a) (b) 


Figure 3.1 (a) Computer rendering of the PNNL 331 -G site; and (b) ASP Test track at the 
Nevada Test Site. 

Because certain masking or shielding materials can interfere with the ability of the 
warning system to detect or identify objects containing special nuclear material (SNM), tests are 
also conducted at NTS with such masking or shielding materials and SNM. Fully integrated 
operational tests follow the field validation tests and also are conducted outdoors at selected U.S. 
ports of entry. 

The committee lias focused much of its attention on performance testing. This is not 
because the other tests are unimportant: Regardless of the performance, the portals will be of 
little use if they cannot operate in real conditions (rain for example ) or if they are incompatible 
with CBP’s computer systems. However, tlie design, execution, and evaluation of these tests are 
comparatively routine, even if solutions to problems revealed by the tests are not. The design, 
execution, and evaluation of performance tests for the portals is more challenging and involves 
more of the science and engineering principles on which the committee lias advice to offer. 

Some types of testing for ASPs are constrained in ways that testing of man)' Department of 
Defense procurement subjects (for example) are not. The main restrictions arise from the DOE 
security regulations for SNM and health and safety requirements. These requirements result in the 
need to separate the testing venues to meet the security needs and not impact health, safety, and 
commerce at operational ports. While it was hoped that later testing would address the criticisms 
of the earlier testing, DHS still lias to operate under the limitations and constraints of security 
required for SNM and minimal impact to the flow of commerce. Furthermore, it is neither possible 
nor desirable to test every possible combination of cargoes and configurations. Physical testing 
with radiation sources, especially special nuclear material, is expensive and time consuming, and 
procurement decisions must be made in a timely fashion. For all of these reasons, the tests need to 
be designed strategically to answer questions about performance across the vast space of possible 
cargo and threat objects, rather than testing that space comprehensively through gross effort. 

As a general principle, the goals of testing and criteria for evaluation need to be clear and 
testable for a test and evaluation program to be effective. In some past testing, the goals and 
criteria were not clear, or they shifted with time. This is one factor that led to test designs the 
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results of which did not adequately answer key questions about performanee. Furthermore, to be 
useful, the goals and criteria need to be relevant. In this case, relevance means that the tests need 
to reflect conditions in real w orld cargo, real environments, and the actual operation of detectors 
in the field. DNDO did base some of its test design on data collected on the stream of commerce 
using a PVT system and an ASP system at NYCT. Much more information relevant to test 
design could have been elicited from data collected on alarms, correlated to shipping manifests at 
ports of entry around the country, even without ASP data. 

One set of goals has been articulated following Congress' language that requires that the 
ASPs demonstrate “a significant increase in operational effectiveness." DHS was responsible for 
defining these terms and in July 2008 issued the definition, found in Sidebar 3.1. The criteria in 
the definition pertain to detection, identification, referrals from primary screening to secondary 
screening, and speed of screening. 


SIDEBAR 3.1 DI IS definition of Significant Increase in Operational Effectiveness of the ASP-C 

Criteria for Significant Increase in Operational Effectiveness [SIOEJ of the ASP-C when deployed for: 
Primary Screening 

If ASP-C satisfies all of the following four criteria for primary screening, then a SIOE has been 
demonstrated, independent of whether the criteria for deployment to secondary' screening have been 
satisfied. These enhancements would increase CBP's capability to interdict SNM as well as reduce the 
volume of traffic requiring secondary' screening. 

1. When Special Nuclear Material [SNM] is present in cargo without NORM, the probability of a correct 

operational outcome for the ASP-C must be equal to or greater than" the PVT RPM. 

2. When SNM is present in cargo with NORM, the ASP-C in primary must increase the probability of a 

correct operational outcome compared to the current end-to-end system as defined above. 

3. When licensable medical or industrial isotopes are present in cargo, the probability of a correct 

operational outcome for the ASP-C must be equal to or greater than the PVT RPM. 

4. When the only radioactive source present in the cargo is NORM, the ASP-C must refer at least 80% 

fewer conveyances for further inspection than the PVT RPM. 

Criteria for Significant Increase in Operational Effectiveness of the ASP-C when deployed for 
Secondary Screening 

If ASP-C satisfies both of the following criteria for secondary' screening, then a SIOE has been 
demonstrated, independent of whether the criteria for deployment to primary have been satisfied. These 
enhancements would increase CBP's capability to interdict SNM while more consistently and 
expeditiously executing sccondaiy screening operations. 

1. When compared to the handheld Radioactive Isotope Identification Dcy'ice (RIID), ASP-C must 

reduce, by at least a factor of two, the probability that SNM is misidentified as NORM, a 
medical industrial radionuclide, unknown, or no source at all. 

2. When compared to the handheld RIID, the ASP-C must reduce the average time required to correctly 

release conveyances from secondary' screening. 

* For HEU, ASP-C must show improved performance compared to PVT RPMs at operational thresholds 
SOURCE: Oxford ct al. (2008) 
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PAST TESTING 


FINDING 

Performance tests prior to 2008 had serious flaws that were identified by the 
Government Accountability Office and the Secretary 's ASP Independent Review Team. 
Tests prior to 2008 did not adequately establish the full capabilities of the ASP systems 
compared with the currently deployed PVT and RIID screening systems, nor whether the 
ASP systems met criteria for procurement. 

This finding is based on several factors, which are discussed in some detail below. In 
briefings to the committee in 2008. DNDO staff agreed w ith several of the criticisms of its prior 
tests and stated that its 2008 tests were designed to address those deficiencies. The 2008 testing 
approach is described in the next section. 

The GAO in 2007 stated that DNDO used biased test methods that enhanced the 
performance of the ASPs: DNDO's NTS tests were not designed to test the limitations of the 
ASPs’ detection capabilities: and DNDO did not objectively test the performance of handheld 
detectors because they did not use a critical CBP standard operating procedure that is 
fundamental to this equipment's performance in the field (GAO 2007b). Specifically, GAO 
wrote “DNDO conducted numerous preliminary runs of almost all of the materials, and 
combinations of materials, that w ere used in the formal tests and then allow ed ASP contractors to 
collect test data and adjust their systems to identify these materials.” 

With respect to bias, the IRT (2008) stated: 

However the IRT’s assessment is that the system's configurations were locked 
and the test results were derived from automated systems that had not been 
modified to benefit from the reduced set of possible outcomes. Operators were 
given no advance guidance on the sequence in which threat objects were 
presented. In short, the IRT did not find any evidence to support the notion that 
the NTS test procedure resulted in the manipulation or biasing of test results, nor 
does the committee believe that the NTS data needs to be discarded on the basis 
of this issue. [Page 91.) 

The committee did not independently verify these facts (e.g.. that the configurations in 
2007 were locked). The committee's understanding of the operational use of the ASP and PVT is 
that the systems provide alarm outputs based on programmed algorithms, not on operator 
decisions, so no intentional real-time biasing of results by test operators was possible during the 
tests. However. DNDO utilized the same sources, masking material, attenuating material, and 
configurations in performance testing that were used in the set up for testing (dry runs and dress 
rehearsals). If the vendors were allowed to calibrate their equipment and adjust their algorithms 
using the test threat objects, then the equipment could more easily recognize the spectra. The 
numbers of sources available were small, but this is not sufficient reason to use the same sources 
for both set up and testing. Device setup and any calibration must use separate sources from 
those used for testing. 

In contrast with the ASP, the RIID requires much more operator interaction. DNDO 
performance tests prior to 2008 did not follow all of the relevant standard operating procedures 
for use of the RIIDs. According to the test plan (DNDO test plan) and briefings to the committee. 
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this error was corrected in the 2008 performance tests. Regarding those procedures, the 
committee observed in visits to ports of entry that the operator actions with RIID and 
Laboratories and Scientific Services (LSS) are inconsistent, which could affect results, and 
would even permit bias- either a positive or a negative bias — for comparing PVT RIID and ASP 
in secondary, although the committee observed no operator bias. Based upon observations at 
operational ports and during the testing at NTS in 2008. even under the best circumstances (ideal 
technical performance by the RIID). the effective use of the RIID depends on the actions of the 
operator and decisions on the spot, which may not be consistent. Die committee observed 
variations in procedure, from one inspection to another, even with the same operator. Ilie 
committee therefore concludes that the RIID is susceptible to ineffective use. 

The committee agrees that pre-2008 tests did not examine the limitations of the ASP’s 
detection capabilities. If all of the results from a particular test are either positive (able to detect) 
or negative (unable to detect), the examiner does not know how close the detector is to the 
transition between positive and negative. The transition can be quite steep, and can be affected 
by other factors that are not controlled in an operational environment. Furthermore, it is useful to 
identify cases in which the ability to detect is poor both because it could help to provide 
guidance on how to improve the system and because there is good reason to believe that 
smugglers will choose smuggling strategies that result in poorer detection. A good physical test 
of the capabilities and performance of a detector system maps the output of the system (the 
result) as one parameter, such as the shielding, is increased stepwise and the detector transitions 
from being able to detect to not being able to detect the radiation of interest. For example, 
according to the IRT review (IRT 2008). the average NORM used in the 2007 NTS tests was 
comparable to the average NORM in cargo observed at NYCT. But a small percentage of cargo 
observed at NYCT had much higher levels, which may be sufficient to mask at least some of the 
threat objects identified by DOF and DNDO. 

SCIENTIFIC RIGOR AND ROBUSTNESS OF DNDO’S 
2008 TESTING AND ANALYSIS APPROACH 


FINDING 

The 2008 performance tests were an improvement over previous tests. DNDO 
physically tested some of the limits of the systems. However, the following shortcomings 
remain. (1) W ithout modeling to complement the physical experiments, the selected test 
configurations are too limited; (2) the sample si/.es an* small and limit the confidence that 
can be placed in comparisons among the results; and (3) in its analysis, some of the 
performance metrics are not the correct ones for comparing operational performance of 
screening systems. 

Many of the flaws in past testing were addressed in 2008 tests. For example, in 2008 
performance tests, real CBP officers conducted the RIID screening of containers referred to 
secondary screening, and DNDO included LSS analysis in evaluating the outcomes of those 
screens. The threat objects (highly enriched uranium and plutonium sources) used in 2008 tests 
had not been used in any previous tests or calibrations, which addressed another criticism of the 
2007 NTS tests. Also, more challenging masking material was used for some cases. Appendix D 
lists the combinations of threat objects, shielding material, and masking material, and their 
configurations used in the 2008 performance tests. 
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However, even with these improvements, shortcomings remain. These include structural 
problems with the testing. 

Without modeling to complement the physical experiments, the selected test 

configurations are too limited 

DNDO was limited by time and resources in what could be evaluated. For example, the 
number and type of threat objects available to the testers through NTS and the Device .Assembly 
Facility (DAF) was small, and only one was the same mass and shape as the objects described in 
the threat guidance. 32 DNDO and its supporting scientists adapted to the lack of a threat source 
that corresponds to the guidance threat by using computer simulations to model the sources and 
determine what mass of threat material in a standard shape w ould emit equivalent radiation. The 
number and type of sources tested cannot be considered “canonical,” i.e.. they do not comprise a 
’‘complete set" from which any possible source in a cargo container can be constructed. 
Although a complete set is not practical or feasible, in the context of modeling described below it 
is likely that a useful subset that spans the space of possible threats can be identified. 

Because the number of possible permutations of cargo material is very large, loading and 
unloading the shipping containers during the tests to cover all possible shielding and masking 
variants is impossible, and the fact that the test sources are only available at NTS precluded the 
assessment of background effects at multiple sites. In light of these limitations, the tests were 
designed to evaluate the response of the detectors to containers with dilTerent configurations: 
empty, a radiation source without additional shielding, a radiation source with shielding, and a 
radiation source with masking material. The test design takes advantage of factorial design, 
which allows for multiple factors to be tested and evaluated at one time, and is considered a 
sound method of experimental design to obtain much information in a limited number of test 
runs (see Appendix C). 33 However, while the test design is reasonable as far as it goes, the tests 
performed are not adequate to fully characterize the instruments nor to predict their performance 
when monitoring the stream of commerce. 

In part to address this problem. DNDO engaged scientists at Pacific Northwest National 
Laboratory, Sandia National Laboratories, the Johns Hopkins Applied Physics laboratory, and 
Los Alamos National Laboratory to carry out “injection studies." These are virtual tests in which 
the gamma spectra of additional test sources, which were experimentally recorded at the national 
labs under controlled circumstances, are added to (“injected into") spectra of cargo in the stream 
of commerce collected by ASPs during the 2007 New York Container Terminal test. These 
combined spectra were then used to challenge the threat identification algorithms of the ASPs. 
For example, of the 22 radiological and industrial isotopes of concern to DNDO. 13 were 
acquired for testing, and nine were considered impractical or unnecessary to obtain for physical 
testing. The response of the detectors to these nine radioisotopes is assessed by “an inspection of 
the threat algorithm" alone. (Description of Medical and Industrial Radionuclides in version 4.10 
of the ASP-C Performance Specification April. 2008) 


32 The committee was told that DNDO selected among the few SNM sources available from the DAF 

33 Practical constraints on the performance testing prevented DNDO from conducting random trials. In other words, 
the same threat object and configuration was passed through the portals repetitively in a linear sequence. Such a 
testing approach is unlikely to detect some kinds of systematic errors, although the committee could not identify 
credible, significant systematic errors that would be missed Randomness is important because the usual methods for 
assigning uncertainties to the results assume random trials and do not account for possible systematic effects. 
However, there are good reasons why these tests could not be random and the committee was unable to identify a 
significant consequence of the non-random tests 
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This type of testing is appropriate, and ealculations of this kind seem to have helped 
DNDO address the problems from 2007. when the performance tests did not chart the performance 
across detection thresholds. The preliminary 2008 test results that the committee has seen suggest 
that the tests found the transition ranges from undetectable to detectable. Hie committee concludes, 
however, that DNDO should go bey ond the existing tests and model a set of test sources that 
represents the spectrum of possible sources and compare the results of the studies to the phy sical 
data acquired during testing to identify flaws in the modeling and algorithms. 

For baseline information. DNDO needs to characterize the performance of the ASP and 
PVT detection systems for the cases of highly enriched uranium, plutonium, uranium-238, with 
and without NORM, and shielding, as well as NORM without threat material. In addition. 
DNDO needs characterization data for the background spectra for non-radioactive containers at 
both NTS and one or more of the representative ports. These data will provide basic detector 
characterization information, which will assist in the development and assessment of 
computerized system models. 

The committee recognizes that the security and health and safety restrictions for using 
SNM in tests preclude doing realistic tests at operational ports of entry and that some 
calculational bridge is needed to explore a detection sy stem's capability. At the time of this 
interim report the committee had not received a full description of the “Injection Studies,” but 
the briefing the committee received indicates that they were done by adding experimental threat- 
object spectra to data collected on actual commerce traffic with NORM present and using the 
algorithms to see what the detection probability would be for the superposed spectra. Hie 
committee would like to see this approach extended to a more robust modeling approach that 
uses simulations of the radiation source, radiation transport through the material in the container 
and to the detector, and the response of the detector to generate the spectrum. These simulations 
need experimental validation and so should be compared to the performance data collected at 
NTS. If they do not agree within statistical uncertainties, then the reasons for disagreement 
should be examined and corrected. When broad agreement has been obtained, then examples of 
observed NORM and medical and industrial radiation sources can be integrated in a model with 
threat material to explore the capabilities of the ASPs and PVTs against a much larger, more 
multidimensional threat space. 

These new simulatioas are distinct from the isotope identification step. DNDO has 
required that the detector systems record data in a standard format, which represents the gamma 
spectrum. Tlie isotope identification software algorithm analyzes the spectrum in that data file. 
Any isotope identification software should be able to analyze the spectrum from any detector and 
from any simulations. There are other important elements of the software, such as reading the 
occupancy sensor and operating the gate arms. Those pertain to integration with the physical 
system, but the isotope identification module is the essential piece for performance of the system 
and is separable from the rest of the system (see Figure 3.2). 
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format 


Figure 3.2 Illustration of the physical system that generates a detected gamma ray spectrum (top) 
and the suggested new modeling to simulate the same process and generate a spectrum (bottom). 
Note: 'Phis drawing is not to scale and docs not show all of the elements or components of the 
detector system. 


To overcome the inherent limitations of physical testing, modeling of the ASP systems 
responses would be invaluublc to the DNDO testing and analysis. With these models, many test 
geometries could be evaluated and the selected results compared to the actual physical tests to 
verify the modeling. Modeling can help to identify configurations for physical testing, and the 
physical tests can be used to validate the models. Accurate modeling could help identify the 
limitations inherent to the technology and the detectors and cun assist in the development of new 
technology over time. 

In the current round of testing, the efTects of shielding und masking were usscsscd 
separately. While this allows for chuructcrixution of instrument response when faced with euch 
scenario, it does not reflect a reulistic scenario in which both masking und shielding mutcriul 
could be used to conccul radioactive material. The effects of the two types of concealment are 
not simply additive, and u combination of the two should be investigated. The number of test 
configurations that cun be tested physically is finite. Loading and unloading of containers with 
shielding and masking muterial is time-consuming, und time spent on testing is costly. 

Here again is u case where u thorough modeling of the well-characterized spectral 
response of the ASP systems would be beneficial in assessing a wider range of scenarios for 
concealment of radioactive material. Data from the shielded-only, masked-only, and shielded + 
masked sources would enable DNDO to assess the validity of the simulations and their ability to 
accurately reflect detector performance capabilities. Using modeling calculations with the 
vendors' algorithms, test scientists can determine configurations of shielding and masking that 


Prcpuhlication Copy 

Copyright © National Academy of Sciences All rights reserved 


1 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00139 Fmt 6601 Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 





134 


CHAPTER 3: TESTING AND ANALYSIS OF THE ASP AND PVT/RHD SYSTEMS 


35 


would likely result in detection and identification in primary and identification in secondary with 
a probability of 50 percent. This would enable DNDO to identify the critical portion of the 
performance curve, that is the transition from correct to incorrect results from the ASP system 
and to confirm these calculations by measurements at NTS. The probability of each outcome can 
be tested at the NTS to confirm the accuracy of the models for select cases and either cause a re- 
evaluation of the models or build confidence. 

'Hie subset of configurations for physical testing to validate models would be chosen to 
test the cases where the expected results, based on simulations, are most sensitive (transition 
regions). In other words, the simulations would be used to predict the configurations that are in 
the detectors’ performance transition (from high-confidence detection to low- or no-confidence 
detection), and the physical tests w ould be run to test that hypothesis. Each set of physical tests 
w ould be used to validate the performance of the models in different regions of the test space. 
Tests that DNDO has already done (including the prc-2008 tests, which used a wider range of 
source materials) could be used in this effort, despite their shortcomings as performance tests. 

Performance testing takes place only at NTS. and DHS’s operational testing of the ASPs 
is planned to take place at only one location: The Port of Long Beach. The committee believes 
that it is important to evaluate the effects of a variation in background intensity and spectra 
because significant variations are expected among the ports of entry across the United States. 
Computer modeling would be able to assist in the identification of limits of the algorithms' 
ability to differentiate threat materials from the background radiation. 

There are many factors that can affect a radiation detector’s capability, but it is not 
possible to test all of the possible variations to threat material configurations, background, 
shielding, and masking within the stream- of-commerce at all ports of entry. 'Hie current 
round of physical testing does not reflect realistic scenarios well, although it does provide 
important information about the response of the detectors to specific, controlled cases. A 
thorough consideration of the methods of concealment of nuclear and radiological material 
that could reasonably be expected from an adversary would better characterize the 
perf ormance of ASPs for the cargo-screening mission. Die models could better cover the full 
lest space of scenarios that need to be evaluated, a goal that cannot be attained practically by 
physical testing alone. 

The sample sizes were small and limit the confidence that can he placed in comparisons 

among the results 

The time and resource constraints mentioned above limited the number of runs for each 
configuration (the sample size) severely: as few as 6 and as many as 12. With such small sample 
sizes, the uncertainties associated with the results are relatively large. This is mostly a concern in 
the performance transition range for the detectors (where the detection probability is neither 1 
nor 0). The number of runs (sample size) for each configuration needs to be large enough that 
the uncertainties (error bars) are small enough for reasonable comparisons to be made to each 
other and to results of simulations. The size of the sample needed can depend on the results of 
the tests. 

In its analysis, some of the performance metrics are not the correct ones for comparing 

operational performance of screening systems. 

Test system performance usually is characterized in terms of detection probabilities, 
measuring the probability that the test system alarms (the test result is positive), given that the 
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screened cargo truly contains threat material, or that it does not alarm (the test result is negative), 
given that the screened cargo does not contain threat material. Because measurement of the 
detection probabilities relies on true know ledge of the cargo contents, one can estimate those 
probabilities only from a designed experiment. 

In real life, however, with real trucks, one observes only the result (alarm status) of the 
screening system. Either the system alarms or it does not, but one does not know the true state of 
the cargo. The result of an accurate system ("alarm" or "no alarm") would be a reliable indicator 
of the cargo contents (SNM or no SNM), but an inaccurate system would be an unreliable 
indicator. One is concerned especially with this question: Given that the test system did not 
alarm, what is the probability that the cargo contained SNM? That is. what risk does CBP take 
by allowing a "no-alarm" cargo to pass? litis "false-negative rate" (FNR) has serious 
consequences. But translating front the measured probabilities to the false-negative rate and the 
false positive rate requires some mathematical manipulation and introduction of an additional 
parameter: the prevalence of threat material in cargo. Given that this parameter is neither known 
nor measurable, comparisons between the performance of two screening systems can best be 
measured by using ratios between the rates for the systems being compared. Such a metric will 
more accurately reflect the relative performance of the screening systems. This issue is described 
in detail in Appendix 13. 

Performance Testing Results and Evaluation 


FINDING 

Because they have large detectors and because of their configuration, ASPs would 
be expected to improve isotope identification, and provide greater consistency in screening 
each container, greater coverage of each container, and increased speed of screening over 
that of the PVT/RIID combination when used in secondary screening. Consequently, tests 
of ASPs in secondary screening are focused on confirming and quantifying that advantage for a 
variety of threat objects, cargos, and configurations. 

The greater consistency, better coverage, and increased speed of secondary screening are 
the results of the configuration of the ASP systems. The ASPs have larger sodium iodide crystals 
than the RIIDs. That size results in higher gamma count rates than in a handheld RIID 
examining the same source, w hich compensates for the greater standoff distance and the shorter 
exposure time for the ASP. The ASPs have better coverage of the containers. The consistency of 
ASP screening depends on the speed of the truck through the portal. As noted elsewhere in this 
report, different CBP officers using the handheld RIID place it differently. Preliminary results 
from 2008 tests confirmed that this is true for the tested cases, but the physical tests could not 
demonstrate that ASPs are superior to the screening system currently in place over the whole 
operational envelope. 

As noted above, when used for primary' screening, an ASP system should be compared to 
the existing combined primary and secondary screening system (both PVT and RIID) because of 
differences in standard operating procedures for primary screening. DNDO’s preliminary 
analysis appears to have accounted for this difference. 

It is not clear to the committee how DNDO w ill interpret the performance test results in 
the context of the criteria for “significant increase in operational effectiveness. Each tested 
configuration is distinct, and averaging across configurations is not meaningful without applying 
normalization or weighting factors. DNDO could use the NYCT data as weighting factors. 
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although there are two challenges associated with this approach: (1) the relevant features are 
multidimensional (gamma flux, radionuclides in cargo, density of attenuating material, 
composition of attenuating material) and (2) NYCT data reflect cargo passing through one large 
port at the time of the data collection, and cargo is different in different ports of entry and 
changes w ith time. Even if these challenges are addressed w eighting factors may only be valid 
for evaluating likely referral rates, not performance against threat objects in containers in 
commerce. Ilie configurations could be weighted according to their frequency in the actual 
stream of commerce (if that could be determined). However, there is no reason to think that 
malefactors will choose the configuration of a cargo container for smuggling a nuclear weapon 
randomly from configurations in the stream of commerce. 

finally, as noted above, there are large uncertainties in the results of these tests. The 
numbers of conveyances for each source were small and the uncertainty associated w ith a small 
sample is large. The costs of conducting larger sample tests with the same number of 
configurations may have been prohibitive, which simply highlights the need to select the 
physical test configurations carefully to maximize the information gained from those tests. 

Operational Testing 

The current plans call for operational testing of the ASP systems that is of short duration 
and limited breadth. ASP systems will be installed at only one site for three weeks. This limited 
testing and subsequent analysis does not allow DNDO to take full advantage of the opportunity 
to collect information about real-world stream-of-commerce effects on detector performance. 
While Pier A at the Port of Long Beach, the location for the test, does have a high volume of 
cargo traffic, it is a location w here the weather generally does not vary a great deal, and the type 
of container coming through the terminal is predictable and not representative of all ports of 
entry (POEs). By limiting operational testing to the environment and the cargo mix at a single 
site, the curtailed field test is missing a prime opportunity to assess detector performance in the 
real world. 

Operational testing is designed to determine if the system is effective and fully useful in 
field, operational settings and when operated by regular users, not just in a laboratory or test 
setting. Operational test and evaluation means the field test, under realistic operational 
conditions, of any equipment item or system intended for use by typical DHS users in defending 
the U.S. homeland: and the evaluation of the results of such tests. Realistic operational testing is 
intended to be independent from the contractor or developer of the system being tested, w ith the 
evaluation of the results also reported independently. 

Realistic operational testing is intended to use production representative systems, 
operated by typical users who may not have the same training or expertise as the scientists and 
engineers who developed the system in the first place. To the extent possible, the system or 
equipment under test is to be operated under realistic stress and operational tempo, in an end-to- 
end manner, using the same procedures as would be expected in everyday use. in an 
operationally realistic environment, with the other interfacing systems with which the proposed 
system is to be interoperable on line. In the case of an RPX1. the “threat” is to be as realistic as 
possible, including both the types of radioactive materials defined in the threat, and the naturally 
occurring radioactive materials that are found in routine commerce. If the system under test 
might be vulnerable to interferences, such as radio communications or other electromagnetic 
interference, those sources should be present in the test also. Finally, because it may not be 
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practicable to conduct a statistically significant number of operational tests, the test challenges to 
the system are to be at the edges of the operating envelope and not only at the center of the 
operating envelope. Contractor involvement in these operational tests is to be strictly avoided to 
eliminate a possible source of bias, the effects of having a highly trained “golden crew'’ 
operating the system, and to gauge the effectiveness of the system when operated by expected 
users. 

At the time that this Interim Report was written, the operational tests planned by DNDO 
had not been conducted, and the committee does not know whether the general guidelines for 
operational testing described above will be followed. 

Changes to the DNDO Approach to Testing 

RECOMMENDATION 

For a more rigorous approach, DNDO should use theory and models of threat 
objects, radiation transport, and detector response to simulate performance, predicting 
outcomes, and use physical experiments to validate or critique the models' fidelity to reality 
and enable developers to refine the models iteratively. With validated models, DNDO can 
evaluate the performance of the ASP systems over a larger, more meaningful range of cases 
than is feasible with physical tests alone. 

To make the testing and evaluation more scientifically rigorous, the committee 
recommends an iterative approach with modeling and physical testing complementing each 
other. .As is noted earlier in the report, the threat space — that is. the set of possible threat objects, 
configurations, surrounding cargoes, and conditions of transport — is so large and 
multidimensional that DNDO needs an analytical basis for understanding the capabilities of 
detectors for screening cargo. DNDO's current approach is to physically test small portions of 
die threat space and to use other experimental data to interpolate within the threat space to test 
the identification algorithms in the detector systems. 

Computer models are essential to the testing process: It is not feasible to examine all of 
die relevant permutations of cargo and threat materials with physical tests alone. Computer 
modeling can examine detector-system and algorithm behavior for a large number and breadth of 
cases with a relatively modest commitment of funds and time. However, die models need to be 
validated against results of physical tests that are carefully designed and selected to represent 
eases covering the test space (the full domain of configurations and compositions of cargo, 
masking material, shielding material, and threat objects). The injection studies that DHS and 
DOE have sponsored enable scientists to test the isotope identification algorithms, but the role of 
injection studies in the overall test plan is still very limited and does not establish an analytical 
basis for understanding the detector systems' capabilities, so a more full and more fully 
integrated approach to modeling and physical testing is needed . 54 


34 GAO describes a PNNL report that discusses the limitations of injection studies 

According to a Pacific Northwest National Laboratory report submitted to DNDO in December 
2006, injection studies arc particularly useful for measuring the relative performance of 
algorithms, but their results should not be construed as a measure of (system) vulnerability To 
assess the limits of portal monitors’ capabilities, the Pacific Northwest National Laboratory report 
states that actual testing should be conducted using threat objects immersed in containers with 
various masking agents, shielding, and cargo. (GAO 2007b) 
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DHS and DOE arc both deploying detectors that screen vehicles and cargo for nuclear 
and radiological material, and both have an interest in better understanding the capabilities of 
deployed and proposed detection systems. The committee recommends that DHS and DOE 
integrate the modeling and testing in a scientific, iterative approach: theory and models would be 
used to predict outcomes of tests; the test outcomes would then be used to validate or critique the 
models; and the models would be used to explore a variety of possible threats, the full range of 
which is very large and cannot be individually tested. 'ITiis kind of interaction between computer 
models and physical tests is essential for building scientific confidence. DOE and its national 
laboratories have extensive experience with both detector development and iterative simulation 
and experimental validation of models, most prominently in the stockpile stewardship program. 
*llie performance tests conducted to date provide some validation points for modeling as well as 
some assessment of detection capability for parameters such as the effects of source, shielding, 
masking, speed, and background radiation level on ASP system performance. These existing 
results are a sensible starting point for validation, but large uncertainties remain in these 
parameters due to limited experimental conditions and small sample sizes. 

For all of the reasons cited above about 2008 performance tests. DHS cannot conclude 
definitively whether ASPs will consistently outperform the current PVT-RIID systems in routine 
practice until the shortcomings are addressed. Better measurement and characterization are a 
necessary first step but may not be sufficient to enable DHS to conclude that the ASPs meet die 
criteria DHS has defined for achieving a “significant increase in operational effectiveness. *’ The 
committee recommends modifications to the current DHS approach to the evaluation procedure. 
These modifications would influence subsequent procurement steps. 

Recommended Approach to the ASP Procurement Process 

RECOMMENDATION 

DHS should develop a process for incremental deployment and continuous 
improvement, with experience leading to refinements in both technologies and operations 
over time, rather than a single product purchase to replace current screening technology . 

In attempting to meet a procurement schedule. DNDO has approached the development 
of the ASP systems as a point goal rather than the beginning of a longer-term process of 
technological improvement. The DNDO approach limits the possibility of iterative 
improvements to the technology' and could result in unnecessary constraints on the ability to 
deploy future nuclear detection systems that would have improved performance characteristics. 


The committee agrees that injection studies and modeling cannot be seen as valid without physical tests with threat 
objects Physical tests are needed for validation, as noted above, but they also can reveal engineering or 
manufacturing flaws Modeling tells how a system should perform, assuming that the equipment as built matches the 
modeled detector, but confirmatory tests are needed with different umls of the same equipment and under different 
conditions The committee's recommendation above states that well validated models can and should be used in 
conjunction with well selected physical tests when it is impractical to do sufficiently comprehensive testing by 
physical tests alone 
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The passive radiation screening of cargo at ports of entry is expected to operate for a lon^ 
time. Although this capability may be enhanced with scanning or interrogation equipment. 3 
Congress has directed CBP to deploy passive detectors as part of the screening procedures for 
cargo entering the United States. CBP has put RPMs in place at hundreds of ports of entry. 

The threat environment, the composition of container cargo, technological and analytical 
capabilities, and the nature of commerce at the ports of entry have changed significantly over the 
last decade and are expected to evolve in both predictable and unpredictable ways in the coming 
years. Containerization changed the nature of shipping in recent decades. Patterns of llow in 
commerce continue to evolve as international trade changes, the world economy adjusts, and 
production shifts among different countries. Patterns of transport also shift in response to costs 
and incentives — for example, rail transport may increase relative to truck transport as pressures 
to reduce carbon emissions and other environmental impacts increase. 

Rather than focusing on the single decision about the deployment of ASPs, the current 
testing should be viewed as a first step in a continuous process of improvement and adaptation of 
the systems. DHS should develop a process for continuous improvement able to address and 
exploit these changes, rather than a single product to replace current screening technology. This 
would enable the system to be updated continuously so that it is not outdated or obsolete by the 
time all of the systems are deployed. 

RECOM MEN DAT ION 

DIIS should deploy its currently unused low- rate initial production ASPs for 
primary and secondary inspection at various sites. This would allow extended operational 
testing with a small investment. 

Such deployment, even on this limited scale, would provide additional data concerning 
their operation, reliability, and performance, and allow DHS to better assess their capabilities in 
multiple environments without investing in a much larger acquisition at the outset. 

'Hie committee has heard DNDO staff say that under current law such deployments are not 
permitted prior to certification. 'Hie committee did not examine this question «ind cannot offer a 
legal opinion, but the committee considers a phased deployment to be a sensible approach. The 
committee recommends that DNDO reexamine the perceived restrictions and. if DNDO concludes 
that such deployments are not permitted, ask for permission to go ahead w ith them. 


RECOMMENDATION 

DIIS should match the best hardware to the best software (particularly the 
algorithms), drawing on tools developed for the competition and elsewhere, such as the 
national laboratories. This should be applied to ASPs and also to improved RIIDs. 

The development of the hardware for radiation detection and the software for analyzing 
the signals from the detectors is separable. It has been useful to have a competitive approach for 
the systems and to see the results. However, as DHS moves forward, it should match the best 
hardware to the best software (particularly the algorithms). In doing so, DHS should draw on 
tools developed for the competition and elsewhere, such as the national laboratories. 


J5 Seaming is a process that actively irradiates the subject with x-rays or gamma rays to generate images of the 
interior of the container Interrogation systems if deployed, would use pulsed neutrons or gamma rays to irradiate a 
container and would alarm on particular radiations from the irradiated cargo 
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The Nal detectors used in the ASP are a mature technology but continued improvements 
in the detection and analysis algorithms can occur with research supported by DOE, DHS, and 
others. The vendors* algorithms are somewhat limited compared to algorithms developed at 
government expense. With data from the hardware in a standard format, it would be 
straightforward to later incorporate new and improved detection and analysis algorithms. 
Further, improved algorithms, or even current .ASP algorithms, could be used to substantially 
improve the performance of handheld RIIDs. 

ASPs will not eliminate the need for handheld detectors with spectroscopic capabilities. 
The greatest deficiency of the RIIDs currently in use is their software. Because some of the 
improvement in isotope identification offered by the ASPs over the RIIDs results from software 
improvements, the best software package should also be incorporated into improved handheld 
detectors. Newer RIIDs with better software might significantly improve their performance and 
expand the range and flexibility of deployment options available to CBP for cargo screening. If 
integration of improved software in hand-held devices is deemed impractical because of the 
computational limitations of a low-power, handheld device, the computational capabilities of a 
handheld device could be replaced or enhanced with a nearby desktop computer system that 
receives data from the handheld detector by wireless transmission. In 2006, DNDO rolled out a 
program to improve RI1D software, called the Human Portable Radiation Detection System 
(HPRDS). However, the committee saw no evidence that this effort was linked to the ASP 
program or that potential improvements in the RIID were being considered in cost-benefit 
analyses (CBA). Linkage makes sense for the technology development, as noted above, and also 
for the CBA. If the HPRDS yields improved RIIDs in the next few years then the ASP 
performance tests w ill have compared the ASPs to outdated technology, which can lead to poor 
choices in cost-benefit tradeoffs. 

By separating the software and hardware elements and engaging the broader science and 
engineering community. 36 DHS would have increased confidence in its procurement of the best 
product available with current technology, and simultaneously could advance the state of the art. 


Correlation of Models and Simulations w ith Physical Test Results 

In addition to operational testing to demonstrate the performance of the system under 
realistic conditioas. one must develop faithful models and simulations to examine scenarios that 
may not have been attempted in the field. The process of validating these models and 
simulations will include predictions of systems performance under conditions that are well- 
defined and can be tested in the field. Only if the models and simulations actually predict 
observed performance under conditions that are amenable to testing (within statistical 
uncertainties) will DHS have confidence that the models and simulations might be dependable 
for describing other configurations. Even then, there may be some configuratioas which the 
models and simulations do not predict adequately. This would not be surprising. To minimize the 
number of potential non-conforming configurations in this set. physical testing needs to explore 
informative, challenging cases. 


34 Even short of the innovation that might arise from broader scientific perspectives, better documentation and peer 
review of the algorithms would make it easier to compare the algorithms and to evaluate this critical part of the 
system. 
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The RPMs must be tested as a complete operational system (not just as components), and 
under conditions that reproduce a fully integrated installation under a range of conditions to 
demonstrate correlation between test results and models and simulations. Similarly, test objects 
must be selected to adequately represent the threat that the system is meant to address. If the 
threat is nuclear terrorism, then the test objects and configurations would include nuclear 
materials in the quantities, shapes, and intensities, along with shielding or masking materials 
designed to foil the RPM. such as might be expected from an inventive terrorist. 

In addition to the improved understanding such testing affords, it can offer operational 
solutions to problems arising from the limitations of the detectors. If the threshold that would 
mask threat objects were known, then all cargo containers that are above that threshold could be 
referred to secondary screening and more thorough analysis. (As noted earlier in this chapter. 
DNDO revised its performance testing for 2008 to address this problem, and preliminary results 
suggest that the tests found the transition ranges.) 

Hie committee believes that by approaching the test, evaluation, and future technology 
development as an iterative process, the limited deployment of the existing ASP systems could 
be a vital tool in improving the technology prior to blanket deployment at U.S. ports of entry. 
Distribution of the existing ASP systems to ports and border crossings in a variety of locations 
and environments (Port of LA/LB. NYCT, and Detroit for example), would provide information 
about the variables in the real-world system that could be fed back into models and could be used 
to develop future generations of the hardware, software, and analytical algorithms. At the very 
least, operational testing should be expanded to take advantage of some of these opportunities. 

Other considerations 

RECOMMENDATION 

Scenaiios identified by red-teaming efforts should be used in developing new models 
and physical tests of detection systems to leani ways of improving the technologies and 
their deployment. 

DNDO already has a red-teaming capability that is applied to operations, and the test 
programs are already intended to identify systematically the detection capabilities of the ASP 
systems. Red teams suggested here as part of an on-going testing mid development program 
could help DNDO (a) identify strategies that smugglers without detailed knowledge of the 
systems are more likely to try and what the adversaries’ adaptation might look like: (b) identify 
new vulnerabilities that the new technologies and CONOPs introduce: and (c) identify what 
technological changes affect the effectiveness of the systems and their applications. Similarly, 
this approach is valuable in test design, ensuring that a realistic range of cases is examined and 
validating the testing protocols, file Special Tests (see Table 3.1) may have served some of this 
function, although they were designed for a slightly different purpose and appear not to have 
been as systematic as what one would expect from a red teaming effort. 

.As noted earlier in this report, DNDO, CBP. and DOE have similar and overlapping 
missions and needs for screening vehicles and cargo. They use and are considering procuring 
much of the same equipment. DNDO has consulted and cooperated with DOE on some aspects 
of the ASP development, but these efforts should be expanded. A wealth of experience dealing 
with algorithm development and archives of data relating to radioactive material and spectral 
analysis exists within the DOE national laboratories. A call to the labs and other agencies for a 
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surv ey of past research and information, assistance, and collaboration could help DNDO tap into 
the expertise within those institutions. 
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Cost-Benefit Analysis 


A well-constructed cost-benefit analysis (CBA) is designed to provide insight about the 
effects of alternative decisions, whether the benefits of a given program exceed its costs, and 
which choices are most cost-effective. This provides a structure for analyzing whether a 
proposed action or program is reasonable and justified. 

As part of the study task, the committee was asked to evaluate the Domestic Nuclear 
Detection Office’s (DNDO’s) CBA of the advanced spectroscopic portal (ASP) technology. As 
of February 2009, the committee had not seen DNDO’s completed CBA, but was provided with 
briefings on the status of the CBA, most recently in October of 2008. At that point, much of the 
information in DNDO’s CBA was described to the committee as still in draft form. There is no 
single definitive approach to doing analysis of such complex cost-risk tradeoffs. The committee 
has chosen to provide suggestions to guide DN1X) in completing its CBA, with the 
understanding that DNDO’s analysis will not be complete until after testing and technical 
evaluation are completed, in the hopes that they will help DNDO make the best case it can for 
each option, but recognizing that the result will still be subject to criticism. 

In addition, the committee hopes that earning out these analyses will lead DHS to 
reexamine assumptions, practices, and objectives to create a firm foundation for continuing to 
improve. For example. DHS ought to consider whether the version of the DOE guidance on 
threat quantities and configurations that DNDO is using is operationally realistic and relevant to 
the threat and the nature of commerce. 

ASSESSING C OSTS AND BENEFITS OF THE ASP PROGRAM 

There could be two bases for saying that a decision to procure ASPs might meet cost- 
benefit criteria: (1) It might lead to a reduction in net dollar costs of procurement, deployment, 
and operation, 3 or (2) it might increase significantly the likelihood of detecting threat materials 
and increase the deterrent value of the systems at a reasonable cost. The preliminary analysis 
presented to the committee by DNDO suggests that the former (criterion 1) likely is not the case 
for ASP deployments. 

FINDING 

Because DNDO's preliminary estimates indicate that the cost increases from 
replacing the PVT/RIID combination with ASPs outweigh the cost reductions from 
operational efficiencies, it is important to consider carefully the conditions under which the 
benefits of deploying ASPs justify the program costs. 


37 In standard accounting practices, costs that arc assigned dollar values arc associated with a relative time of 
expenditure for comparison and discounted appropriately to give net present value, or some other, similarly 
consistent discounting method is applied It is the committee’s understanding that DNDO is following standard 
practice for discounting 
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When, where, and how terrorists might attempt to use a nuclear weapon is unknown. 
This uncertainty makes it difficult to decide whether or not to invest in a system for detecting 
nuclear w eapons. The consequences of a successful nuclear detonation by a terrorist group could 
be catastrophic. The potential magnitude of the consequences is a major factor in justifying 
investments to reduce the risks from nuclear scenarios. Hie risk of such scenarios depends also 
on how likely it is that they might occur. However, because the likelihood of such an attack 
cannot be precisely specified it is difficult to estimate the risk from nuclear weapons and the 
extent to which this risk is reduced through defensive countermeasures. Despite this uncertainty, 
a structured CBA can help to guide decision-making in such a situation. 

It is important to consider the standards used to measure ASP performance and whether 
meeting these standards is sufficient to warrant the program expenditure. A well-const meted 
cost-benefit analysis can aid in evaluating whether the criteria in DHS's definition of a 
“significant increase in operational effectiveness*’ (shown in Sidebar 3.1) add benefits sufficient 
to justify the cost of the ASP program. 

FINDING 

DIIS' definition of “significant increase in operational effectiveness” is a modest set 
of goals: The increases in operational efficiency do not by themselves appear to outweigh 
the cost increases from replacing the PVT/RIID combination with ASPs, based on DNDO’s 
preliminary estimates, and the criteria do not require significantly improved ability to 
detect special nuclear material in primary screening (see Sidebar 3.1). 

If the ASPs meet the defined criteria and are able to detect the minimum quantities of nuclear 
threat material that DOE recommends (DOE guidance). DHS still will not know whether the 
benefits of the ASPs outw eigh the additional costs associated with them, or whether the funds are 
more effectively spent on other elements of the Global Architecture. 

In particular, to determine whether the benefits outweigh the costs, the following issues 
need to be addressed: 

• The relative effect of the ASP system, relative to the existing PVT/RIID system, on 
reducing the probability that an adversary would try to smuggle nuclear material into the 
United States (deterrence); 

• The relative effect of the ASP system, relative to the existing PVT RIID systems, on the 
probability that an adversary w r ould succeed in smuggling nuclear material into the 
United States. 

• Whether any benefits identified in the above effects assessments and the improvements 
required by the SIOE criteria merit the cost of the improved technology deployment; 


RECOMMENDATION 

The CBA should provide a convincing narrative involving all relevant costs and 
benefits in order to justify spending funds on the ASP program. 

Many of the costs and particularly the benefits involved in threat detection systems such 
as the ASPs are not easily quantifiable. However, the CBA should provide a convincing 
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narrative involving the relevant costs and benefits in order to justify spending funds on the ASP 
program. This narrative should provide qualitative justifications and explanations where 
quantitative justifications are not available. It may be difficult to justify a threat detection 
program based only on easily quantifiable benefits such as such as the benefits of reduced man- 
hours spent scanning cargo. However, a major benefit of threat detection programs is an 
increase in security, and careful consideration needs to be given to addressing these benefits in a 
thorough way. The committee provides some guidance to DNDO on addressing security benefits 
later in this chapter. 

In a structured CBA. several key elements need to be thoroughly addressed, 
quantitatively where possible, and qualitatively where necessary, as discussed above. These key 
elements have been defined by the United States Office of Management and Budget (OMB, 
2003) as: 

• A clear statement of the objectives of the program, w hich would address what the ASP 
technology is meant to accomplish relative to the polyvinyl toluene (PVT) technology 
and how it fits into the rest of the Global .Architecture (defined in Chapter 1). 

• An assessment of meaningful alternants, which would address a full range of reasonable 
options and the benefits of the ASP program relative to these options, including a good 
baseline (typically a no-action alternative). 

• A comprehensive, credible and transparent analysis of benefits and costs as appropriate. 
which would address a full range of qualitative and quantitative benefits, including 
security benefits, as thoroughly as possible. 

In the following sections, the committee offers guidance to DNDO in these three areas. 

STATEMENT OF THE PROGRAM OBJECTIVES 
RECOMMENDATION 

A cost-benefit analysis should dearly define the ASP program objectives, including: 

• Describing the new and unique capabilities of the ASPs in the context of their role in 
the Global Architecture; and 

• Defining a realistic baseline alternative against which to compare the ASP 
deployment. 

A structured CBA begins with a clear description of the objectives of the program and the 
specific needs it is designed to meet. For the ASP program, the committee judges that the major 
issues to be addressed when stating the program objectives are: 

• Describing the unique and new capabilities of the ASPs that will enable them to meet 
the program objectives and clarifying their role in the Global Architecture: and 

• Defining a realistic baseline scenario, to give the program full credit for benefits and 
costs. 

The following sections discuss some specific recommendations from the committee for 
DNDO's future w ork that concern a clear statement of the program objectives. 
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Describing new and unique capabilities and the role in the Global Architecture 
REC OM MEN DAT ION 

The larger context of global security should be considered in the ASP program's 
cost-benefit analysis. In particular, DHS should consider tradeoffs and interactions among 
different elements of the Global Architecture. Alternative approaches can be used to 
prevent the smuggling of nuclear materials into the United States. Some are alternative 
approaches to the cargo screening problem, while others are outside the program scope 
(such as prevention of smuggling via mil, aircraft, or small boats). 

It is important to show clearly the new and unique capabilities that deploying the ASPs 
will provide, relevant to stated goals and operational outcomes. In particular, this will involve 
considering how the ASP's capabilities contribute to the larger context of the threat detection 
system intended to prevent nuclear and radiological threat material from entering the I’nited 
States, known as the “Global Architecture." Radiation portal monitors (RPMs) detect threat 
materials entering the I’nited States in cargo containers on trucks via land border crossings and 
seaports, and constitute one piece of this system. The Global Architecture also encompasses 
screening for nuclear threat material brought across U.S. borders by plane, by personal 
watercraft, by rail, or that is transferred promptly to on-dock rail cars at seaports. In October 
2008. DNDO presented the scope of its CBA as limited to the then-current deployment plans, 
which included ASP-C 38 (land and sea cargo) and ASP-D (the wide-load variant) but excluded 
rail deployments. 39 The committee had not seen any more recent information regarding the 
scope ofDNDO's CBA as of the writing of this report (February 2009). 

There are tradeoffs that need to be considered among the many programs that make up 
the Global Architecture. Given the limited resources available, investment used to strengthen the 
Global Architecture can be applied towards: (1) using different (newer) technologies to fill the 
same gap: or (2) filling different gaps, for example, different threats, different geographies, and 
different modes of transport. Furthermore, the preferred modes and routes of shipping and 
transportation are not static. Nor are threats. A more comprehensive evaluation of security 
benefits would factor in such trends For example, the enhanced capabilities provided by the 
ASP-C system are relevant to cargo containers entering the United States by truck, but not by 
rail. In the future, it is probable that less cargo will be brought directly into major U.S. seaports. 
Some of the fastest-growing ports in North .America are in Canada and Mexico. 40 and it is 
expected that these ports will handle increasing amounts of cargo destined for the United States. 
Much of this cargo will be unloaded onto on-dock rail and will cross U.S. borders on rail. 

The tradeoff's among different spending options need to be considered in the ASP CBA or 
in a higher level analysis about allocation of efforts and funds. Indeed, it may be more 
appropriate for such an analysis to be carried out at a higher level so that it can provide guidance 
and support for multiple programs in a coordinated fashion. The committee has. however, seen 
no evidence that the higher level tradeoff' analysis has been done, and a recent report by the GAO 


38 Anything that is legal to drive on a road can pass through the ASP-C variant 

39 The ASP-D is considered a minor variant, however. DNDO informed the committee in October. 2008 that 
because the modifications needed to accommodate on-dock rail were significant, they would not be including on- 
dock rail as part of the program scope. 

40 Some of the fastest-growing ports in North America include Manzanillo. Lazaro Cardenas, and Vancouver For 
further information, sec http /'aapa.filcs cms-plusxom/PDFs/North_ American Container Traffic pdf 
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(2009) tends to confirm this view. Furthermore, an ASP CBA should at the very least consider 
alternatives arising from work in the same office on the same mission (e.g.. improved RIIDs). 

Defining a good baseline 

The benefits and costs of a program are defined in comparison with a clearly stated 
baseline, typically a “no-action" alternative. This is the type of baseline DNDO indicated to the 
committee that they had chosen as of October 2008: In this baseline, no ASPs would be 
deployed, and PVT deployment would be expanded. PVTs would continue to be used in both 
primary and secondary screening, and the handheld RIIDs would be used in secondary screening 
according to Custom and Border Protection's (CBP) current Concept of Operations (CONOPS). 

It is important that the baseline option reflect the key features of the actual systems as 
they are deployed today, to ensure that costs and benefits are being accurately assigned to the 
ASP program. In considering DNDO’s baseline scenario, one must take into account several 
assumptions about the current operations of the PVTs to give the ASPs appropriate benefit for 
providing increased security. In this case, a good baseline might include: 

• Using the actual sensitivity settings of PVTs deployed in various ports: and 

• Correctly accounting for the range of densities of typical materials in containers. 

.As of DNDO’s October 2008 presentation to the committee, the baseline alternative in 
the CBA presumed that the PVTs at all ports were set to the same detection level and operated 
identically. As discussed earlier in this report, the PVTs signal a “detection" when the observed 
radiation exceeds a given level. When this occurs, the conveyance is pulled aside for a 
secondary screening using a second PVT (the truck passes through the detector at a slower rate) 
and a hand-held RIID operated by a CBP officer. However, cargo containing NORM (for 
example, granite countertops or porcelain toilets) can set off these detectors, which are unable to 
distinguish between NORM cargo and threat materials. If the PVT is set more sensitively (the 
system alarms when it detects lower levels of radiation) then more NORM cargo is diverted to 
secondary screening than if the PVT is set less sensitively. 

If the alarm threshold is set too low (the system is too sensitive), the flow of commerce 
can be affected by the amount of NORM cargo sent to secondary screening. If the alarm 
threshold is set too high (the system is not sensitive enough) then material of concern could pass 
through without an alarm. More background regarding thresholds and sensitivity can be found in 
Chapter 2. To the extent that the ASPs allow detectors to be operated with a greater level of 
sensitivity, they would be expected to detect threat materials more reliably. 

In addition, the baseline needs to recognize and account for (to the extent possible) the 
range of material densities in typical containers that is brought into a given port. Threat material 
could be shielded by cargo brought into the United States in ordinary commerce. An 
operationally realistic range of material densities can then be used to define the range for 
comparative evaluation between the ASPs and the PVT RIID systems. At present, the DOE 
threat guidance uses a single value for cargo density, and that value does not represent the upper 
limit of the range of typical cargo densities. Lower density cargo provides less shielding and 
therefore a less challenging detection problem. 
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ASSESSMENT OF A RANGE OF MEANINGFUL ALTERNATIVES 
RECOMMENDATION 

A cost-benefit analysis for the ASP program should demonstrate that a full set of 
meaningful alternatives has been assessed, including alternative deployments, operations, 
and technologies. 

A well-performed CBA demonstrates that a full set of meaningful alternative approaches 
to the proposed program has been assessed relative to the baseline. According to OMB Circular 
A-4 (OMB, 2003), a well-performed analysis ”describe(s] the alternatives available and the 
reasons for choosing one alternative over another ... [i]t is not adequate simply to report a 
comparison of the agency ’s preferred option to the chosen baseline.” 

DNDO informed the committee in October of 2008 that three potential deployment plans 
for the ASPs are currently being considered, in addition to a baseline plan, discussed above. 
These three plans include: 

1 . Deployment in secondary v expand the deployment of PVTs in primary screening, and 
deploy ASPs in secondary screening; 

2. Deployment in primary’ and secondary ; deploy ASPs in primary screening and 
secondary screening; and 

3. Hybrid deployment: For primary screening, deploy some primarily in high traffic 
ports, and retain PVTs in other, lower-traffic ports. For secondary screening, install 
ASPs at all ports. 

The committee was not shown assessments of other alternatives (apart from the baseline 
alternative) as of February 2009. although DNDO may have analyzed others. Increased security 
might be achieved without the deployment of ASPs, and a good CBA would clearly demonstrate 
that all reasonable possibilities have been assessed, or give reasons why these possibilities were 
not assessed. 41 

Several alternative approaches are possible. The comparative costs and benefits of 
changes to CBP’s CONOPS could be assessed. For example, detection equipment could be 
deployed at every exit and CBP could select random nuisance alarms to be examined, potentially 
resulting in heightened deterrence effects; or secondary scanning times could be increased, 
providing time for high purity germanium (IIPGe) detectors to perform Identifications. .Another 
alternative could be to maintain current CONOPS, but deploy alternative technologies, such as 
improved RIIDs. The use of newly available RIIDs and associated software could improve the 
performance of secondary screening: newer, more sensitive models of RIIDs are available. 
Alternatively, software on the existing RIIDs could lie improved. 4 This list is meant to be 
illustrative and not comprehensive. DNDO and CBP may have insights into alternatives of which 
the committee is not aware. At the same time, for practical reasons, the set has to be finite and 
relatively small, so the cases should be chosen carefully to represent the most promising 
alternatives. 


41 In addition, note that CBP officers may prefer the use of handheld detectors, and some deployment of RIIDS may 
be needed even in the case that the ASPs are deployed in secondary 

41 For example, RIIDs using high purity germanium detectors are available Improvements to RIIDs and to the 
associated software arc discussed in more detail in Chapter 3 of this report 
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A COMPREHENSIVE, CREDIBLE, AND TRANSPARENT ANALYSIS 
OF BENEFITS AND COSTS 


RECOMMENDATION 

A cost-benefit analysis for the ASP program should include a comprehensive, 
credible and transparent analysis of benefits and costs, although the committee 
acknowledges that DNDO will not be able to perform a full, quantitative cost-benefit 
analysis. 

DNDO’s CBA is intended to provide guidance about whether the additional project costs 
associated with deploying the ASP systems are outweighed by the improvements in detection 
and other benefits. In October of 2008. DNDO presented a preliminary set of total life cycle cost 
estimates (LCCE) for the three ASP deployment scenarios outlined previously as well for a 
baseline scenario. However, the committee did not see a breakdown of these costs into 
categories such as design and development; procurement; deployment; maintenance; operations; 
or decommissioning, so it is unable to assess the validity of the projected costs. The approach 
that DNDO described to the committee for the ASP LCCE appeared to use a reasonable 
methodology. The committee did not see the details of this assessment; however, they suggest 
that it is essential to a valid CBA to supply uncertainties associated with the projected costs. The 
cost assessment should cover all phases of the acquisition life cycle in a manner that is 
independent of contractor or program office biases and assess the risk of cost escalation 
associated with the estimate. It is also possible that adoption of new technologies will lead to 
cost reductions, although this is less common in such procurements. 

As of the writing of this report (February 2009), DNDO had not yet presented an 
assessment of the security benefits of the ASP program. DNDO had considered some benefits, 
such its the ability for CBP officers to be reassigned to other missions, time saved in secondary 
screening, and a reduced number of conveyances referred to secondary screening. However, 
according to DNDO (2008b) these benefits alone are unlikely to justify the costs of the ASP 
program, and other, more difficult to quantify benefits of will need to be taken into account, 
including security benefits, This is a point on which DNDO has asked for specific advice from 
the committee. 

Hie committee recognizes the likely inability of the DNIX) (or anyone else) to perform a 
full, quantitative cost-benefit analysis for the ASP program. Despite these difficulties, at 
minimum, a logical connection of the program effort to its goals needs to be presented. 

A well-performed CBA that helps in procurement decisions for ASPs is not going to be a 
simple analysis following standard formulae commonly used in other kinds of procurements. 
DNDO expressed its difficulty in assessing two of the cost-risk elements in the cost-benefit 
analysis with respect to equipment performance: assessing the probabilities of failure to detect 
threat material, and factoring in potential consequences of such a failure. There are four 
probabilities involved in analyzing security benefits: Pdaect, or the probability of detecting the 
threat material; Pideniification* or the probability of correctly identifying the threat material; P interdict- 
or the probability of interdicting the threat material; and Pencoumer, the probability of an adversary 
attempting to smuggle threat material into the United States in the first place. The last probability 
is likely to be highly complex to evaluate, and indeed impossible to determine definitively. 
Although it is difficult to estimate such probabilities with high confidence, analysts need to 
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understand what they can about these probabilities based on analytical tools and input from the 
intelligence community and other sources. 4 

The consequences are likewise uncertain, although in this case because of indeterminacy. 
The consequences of a successful nuclear or radiological attack could be assessed within 
reasonable uncertainty bounds if the nature of the weapon, its location, and all of the 
environmental conditions, including where the people are (e.g., is it rush hour?), were specified. 
But in a site-generic, time-independent assessment, the variability in the possible consequences 
is very broad. All of these factors make it quite difficult to consider avoided events in the cost- 
benefit analysis. However, that difficulty does not make it less important to consider these 
factors. 

Security benefits can result from changes in any of the above probabilities. The benefits 
can take the form of higher detection, identification, and interdiction probabilities. They can also 
result from deflection (e.g. to overseas targets) or deterrence (effectively reducing the probability 
of encounter), so the security benefits associated w ith these factors also need to be considered. 
(Note, however, that deflection can push adversaries to use different avenues to the same target, 
as explained below.) The existence of some radiation monitoring at seaports and land border 
crossings may provide sufficient discouragement to potential adversaries from smuggling via this 
route. However, it may also increase the probability that the terrorist will focus on other gaps in 
the nation's security that are identified as easier targets. For these reasons, benefits from 
increased detection probabilities may be modest as long as there are significant gaps in the 
Global Architecture. Improved detection can be expected to become more beneficial as those 
gaps are filled. 

There are several analytical approaches that may be useful to DNDO in performing an 
analysis of the security benefits of the different alternatives proposed for the .ASP program. 
Below- are three examples. Each of these options suffers a common shortcoming - they do not 
answer the question of whether the benefits of implementing ASP exceed the program’s costs. 
However, each in a different way can provide insights that could help the Secretary' weigh the 
merits of acquiring and deploying ASPs or alternative nuclear detection technologies. 

A capability-based planning approach would provide a structured assessment of how 
alternative detection technologies or deployment strategies reduce the risk of a nuclear 
detonation in the United States. This approach has been applied in defense applications to 
compare and contrast a set of options for approaching a given operational challenge across a 
wide range of circumstances (Davis 2002). In the case of ASP. the operational challenge is to 
prevent a nuclear or radiological attack by detecting, identifying and interdicting materials or a 
weapon smuggled into the United States. The set of options could include alternative 
deployments of ASPs and PVTs. deployment of alternative RIIDs, or (depending on the scope of 
analysis) shifting emphasis between port-of-entry (POE) and non-POE detection. The 
circumstances considered would include relevant dimensions of the adversary capability and 
tactics, the operating environment, and the technologies themselves. For example, circumstances 
could include different types of nuclear materials, use of different shielding or masking methods, 
technology performance now or in the future, different operating environments, different 
numbers of weapons available to the adversary, or failure of primary and secondary inspection 


43 Other parts of DHS have conducted expert elicitations of threat probabilities with members of the intelligence 
community A recent report by the National Research Council cautioned about limitations of this approach for the 
bioterrorism risk assessment The committee, however, secs value in factoring in what the intelligence community 
knows and suspects, accounting at the same time for the confidence that can be placed in that knowledge 
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due to a common source of failure. Using a model that reflects the detection systems 
performance, the alternatives are then compared across multiple metrics appropriate for these 
circumstances (e.g., probability of detection of a representative threat object. The strength of a 
capabilities-based planning approach is that it can provide a rich comparison of the security 
benefits emphasizing the circumstances under which each option might be preferred. The 
weakness of this approach is that exploring the circumstances that affect the systems capability 
can quickly lead to large and complex analysis and judgment of the analysts is required to 
balance this complexity against and ability to draw salient insights about a systems capabilities. 

Game theory could provide insight into the benefits from deterrence associated with the 
PVTs and ASPs. Studies in other areas have found that the simple presence of security can 
significantly deter individuals from choosing the action that the security is meant to protect 
against. For example. Ayres and Leavitt (1998) used game theory to predict that the ability of a 
criminal to observe security measures affects the deterrent ability of that security measure and 
validated this prediction with observations from vehicle theft statistics: An increase in the 
percentage of lojack-equipped 4 4 vehicles in a given area is associated with a substantial decline 
in auto theft. 45 In contrast, observable security devices against car theft tended to merely shift 
the risk of theft to other vehicles, but not lower overall rates of theft. In another example of 
game theory being used for security policy, Kunreuther (2005) demonstrated the public policy 
opportunity that exists in commercial aviation because of tipping effects that would lead to mass 
adoption of baggage screening technologies under the right policy incentives. Researchers who 
have used game theory to assess protection of critical infrastructure from terrorism are just 
beginning to explore the utility of these approaches to decisions about radiation portal monitors 
(Bier and Azaiez 2009; Dighe et al. 2009). as noted earlier. In the context of ASP procurement, 
one would have to look at the incremental benefit of installing new detectors. The general 
weakness of game theory is that if analysts are unable to estimate parameters of their model, they 
are only able to draw broad and conditional conclusions about adversary' behavior. F.xamples of 
parameters from security applications that are difficult to estimate include the value to the 
adversary of different outcomes (i.e., what might constitute successes and w r hat are the costs of 
being caught), the probability of attack, and the costs of the defender falsely suspecting an attack 
is occurring. In the absence of being able to establish values for model parameters, broad or 
conditional conclusions might or might not provide actionable advice to policymakers. 46 

Finally, cost-effectiveness analysis and break-even analysis are related approaches that 
have been used to assess costs and benefits when performing a complete cost-benefit analysis is 
difficult or impossible. Cost-effectiveness analysis is used w r hen valuation of benefits is 
contentious and it is not possible to quantify benefits in monetary terms. Because the security 
goals of the ASP program may be difficult to value monetarily, comparing program alternatives 
using cost-effectiveness measures such as dollars per life saved or dollars per attack avoided 
could provide insights into their relative merits. In contrast, break-even analysis can be used 

44 Lojack is a hidden radio-transmitter device used for retrieving stolen vehicles 

45 This occurs without a drop in other types of crimes. (Ayres and Leavitt, 1998) 

46 Consider the case of deterrence Partial screening (screening a fraction of the total number of containers entering 
the United States) may provide for effective deterrence (or deflection), if detection probabilities arc sufficiently high 
and if smugglers cannot predict which containers will not be screened This benefit quickly evaporates if adversaries 
arc able to stage several smuggling attempts simultaneously because the chance of at least one attempt succeeding 
grows rapidly with the number of attempts (Bier and Haphunwat 2009). Like other game theoretic analyses, this 
enters a psychological realm, ascribing logical values to the adversary (e g . that the threat material is a scarce and 
valuable asset and that the risk of discovery at a port of entry is not desired). 
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when it is difficult to even assess the magnitude of benefits in any units. Break-even analysis 
determines conditions that must be met for benefits to exceed costs. In security applications, 
these conditions could be a required reduction in overall risk (see Willis and LaTourrette 2008 
analysis of the Western Hemisphere Travel Initiative) or a baseline estimate of a threat of attack 
that exists (see Martonosi et al. 2006 analysis of 100°o container inspection). In some instances, 
framing the problem in this manner has proven useful because it also can provide a simple, yet 
fully parameterized model of the system being evaluated which a policymaker can explore to 
understand conditions that must be met for benefits to exceed costs (see von Winterfeldt and 
O'Sullivan analysis of acquisition of MANPADs defenses on commercial aviation). In cases 
where break-even analysis identifies meaningful bounds on decisions, that is threshold 
conditions that can easily be judged to exist or not exist, this approach can simplify 
decisionmaking. The downfall of break-even analysis is that these conditions do not always 
exist. 

The committee reiterates that methods for evaluating security benefits, examples of 
which are provided above, can provide different insights based on their approach, and none is 
likely to provide fully quantitative and definitive results. But most policy decisions are made 
without fully quantitative and definitive results, so DNDO should incorporate these benefits to 
provide the most informative CBA it can. 

The committee recommends that DHS not proceed with further procurement until it has 
addressed the findings and recommendations in this report and the ASP is shown to be a favored 
option in the cost-benefit analysis. 
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Appendix A 


The Joint Explanatory Statement and the Statement of Task 


In the Joint Explanatory Statement for the 2008 Consolidated Appropriations Act (P.L. 110-161), 
Congress stated the following: 

The Committees on Appropriations appreciate the difficulties the Secretary faces in 
certifying the ASP systems and provide sufficient resources to allow DNDO to enter into 
an agreement with the National Academy of Sciences (NAS) to assist the Secretary in his 
certification decisions. NAS will help validate testing completed to date, provide support 
for future testing, assess the costs and benefits of tills technology, and bring robustness 
and scientific rigor to the procurement process. 

Working with the Domestic Nuclear Detection Office, the National Research Council, the operating arm 
of the National Academy of Sciences, developed the following statement of task for this effort. 

The chairman of the National Research Council will appoint a committee of experts to 
perform tasks addressing the Secretary of Homeland Security's requirements for certification 
of advanced spectroscopic portals (ASPs) for secondary screening and, to the extent possible, 
for primary screening. The committee will evaluate the Domestic Nuclear Detection Office's 
(DNDO’s) ASP assessments, performance tests, and analyses. Specifically the committee will 
Evaluate the adequacy of the DNDO's past testing and analyses of the ASP systems; 
Evaluate the scientific rigor and robustness of DNDO's testing and analysis approach; 
Evaluate DNDO’s cost-benefit analysis of ASP technology. 
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Appendix B 

Performance Metrics for ASPs and PVTs 

"Far better an approximate answer to the right question, which is often vague, than 
an exact answer to the wrong question, which can always be made precise. " 

John W. Tukey (1962). “The Future of Data Analysis." Annals of Mathematical 
Statistics 33(1), p. 1-67. (The citation appears on p. 1 2.) 

When evaluating the performance of instruments to identify the system most well suited 
to a given task, one needs to consider the correct metric for making the comparison. In the case 
of systems such as the Advanced Spectroscopic Portals (ASPs), conventional measures such as 
sensitivity and specificity provide useful information, but do not assess directly test performance 
in actual field operation. The metrics of interest concern the probabilities of making incorrect 
calls ~ i.e.. the probability that the cargo actually contained dangerous material when the test 
system allowed it to pass (a false negative call), and the probability that the cargo actually 
contained benign material when the test system alarmed it (a false positive call). In some 
contexts, the false negative call probability (FNCP) has been called the "false non-discovery 
rate” and the false positive call probability (FPCP) has been called the false discover*' rate" (see 
Note 1). This appendix describes the calculations leading to estimates of these probabilities, the 
uncertainties in these values, and how these estimated probabilities can be used to compare two 
systems under consideration. 

Test system performance usually is characterized in terms of detection probabilities. The 
notation for these probabilities comes from the literature for comparing medical diagnostic tests, 
and we use the same notation here for radiation detection systems: 

• Sensitivity (S) probability that the test system alarms, given that the underlying cargo 
truly contains special nuclear material (SNM) 

• Specificity (T) = probability that the test system does not alarm, given that the underlying 
cargo truly contained benign material (non-SNM) 

• Prevalence (p) probability that cargo contains SNM 

• Positive predictive value (PPV) = probability that the underlying cargo truly contains 
SNM. given that the test system alarms 

• Negative predictive value (NPV) = probability that the underlying cargo truly contains 
non-SNM. given that the test system did not alarm. 

Because the definitions of sensitivity (S) and specificity (T) rely on true knowledge of the 
cargo contents, we can estimate a system’s sensitivity (S) and specificity (T) only from a 
designed experiment. The experimenters insert into the cargo either SNM (true SNM) or benign 
material (true non-SNM). and then run the cargo through the test systems; the proportion of 
(true-SNM) runs that properly set off the test system alarm is an estimate of the test’s sensitivity, 
and the proportion of (benign-SNM) runs that properly pass the test system is an estimate of the 
test’s specificity. 

In real life, however, we do not know the cargo contents. We see only the result of the 
test system: either the test system alarmed, or it did not alarm. Operationally, if the system 
alarms. SNM is suspected; if the system does not alarm, the cargo is allowed to pass. We are 
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concerned especially with this question: Given that the test system did not alarm, what is the 
probability that the cargo contained SNM? That is, what risk do we take by allowing a “no- 
alarm” cargo to pass? From the standpoint of practical operational effectiveness, this probability 
(the probability that the cargo contains SNM, given that the test system did not alarm) has grave 
consequences. As shown below by Baves’ Theorem, it is a function of sensitivity (S) and 
specificity (T), as well as of prevalence (p) (i.e., how likely a positive - here, a cargo containing 
SNM - is likely to occur), but a comparison between two test systems on the same scenario (i.e.. 
the same threat) involves the same prevalence, so prevalence does not enter into the comparison 
of effectiveness for the two test systems. So accurate estimation of sensitivity (S) and specificity 
(T) is important, in that it allows us to compare accurately the performance of two test systems 
using the relevant, practically meaningf ul metric. 

The probability of making a false negative call (FNCP) is the probability that the cargo 
truly contains SNM. given that the test system did not alarm; it is exactly the same as 1 - NPV. 4 
Unfortunately, we cannot estimate NPV from real life runs of the radiation test system, because 
in real life, we don't know the true state of the cargo. We can. however, estimate S and T from 
designed studies, such as those conducted at the Nevada test site, because we know the cargo 
contents in the tests. We also can derive confidence limits on S and T from such designed 
experiments, and hence we can estimate (1 NPV) and associated confidence intervals. More 
importantly, we can compare the two systems via a ratio, say ( 1 -NPV ] )/( 1 -NPV,)', a ratio whose 
lower confidence limit exceeds 1 indicates preference for test system 2, while a ratio whose 
upper confidence limit falls below- 1 indicates preference for test system 1. Note that these ratios 
may differ for different scenarios; a table of these ratios may suggest strategies for associating 
the ratios with the threat levels presented by different scenarios. 

Notice also that the probability of making a false positive call (FPCP) is likewise of 
interest for purposes of evaluating costs and benefits: too many false positive calls can also be 
costly (e.g.. slow-ing down commerce, diverting CBP personnel from potential threats as they 
spend time investigating benign cargo, etc.). Two detection systems that have exactly the same 
probability of a false negative call (1-NPV) for a given scenario, but substantially different 
values of the probability of making a false positive call, may indicate a preference for one system 
over the other. The probability of making a false positive call equals 1 PPV. 

We illustrate these calculations from hypothetical data below. Suppose we have 24 
trucks, into 12 of which we place SNM and leave only benign material in the remaining 12 
trucks. We run all 24 trucks through two test systems, and observe the following results: 


Test System 1 Test System 2 



Alarm 

No 

Total 

Alarm 

No 

Total 


Alarm 

Runs 

Alarm 

Runs 

SNM in 
cargo 

10 

2 

12 

a 

i 

12 

Non-SNM 

in cargo 

4 

8 

12 

2 

10 

12 


14 

10 

24 

13 

11 

24 


47 The literature (see references) refers to "false discovery rate” and "false non-discovery rate” which are related 
to (1 -PPV) and (I -NPV), respectively, but their definitions are slightly different (sec Note 1). 
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Sensitivity is the probability that the system alarmed, given the presence of SNM in the 
cargo: among the 12 trucks that contained SNM, 10 alarmed for test system 1 (estimated 
sensitivity 5, = 10/12) and 11 alarmed for test system 2 (estimated S 2 = 11/12). Similarly, we 
estimate specificity for the two test systems as 8/12 and 10/12, respectively (number of “no 
alarm” results out of the 12 non-SNM trucks). Because we specified the number of runs in each 
condition (/?,= 12 for SNM runs and « 2 =12 for non-SNM runs), we can estimate the 

uncertainties in these probabilities using the conventional binomial distribution. In this case, 
lower 95® o confidence bounds determined from the binomial distribution based on fl, = n 2 =12 
are: 

Test System 1 Test System 2 

Estimated Sensitivity 0.833 (10/12) 0.917(11/12) 

95% confidence interval (0.562,1.000) (0.661,1.000) 

Estimated Specificity 0.667 ( 8/12) 0.833 (10/12) 

95 % confidence interval (0.391, 1.000) (0.562, 1.000) 

(The wide intervals result from the small sample sizes.) 

More importantly, the negative predictive value (NPV. the probability that the truck truly 
did not contain SNM. given that the alarm did not sound) is 8/10 for test system 1 and 10/1 1 for 
test system 2. and hence we estimate the probability of making a false negative call for the two 
systems as 

• proportion of cases where test system 1 did not alarm (10 cases) but actually 
contained SNM cargo (2 cases) = 2/10 = 0.20 

• proportion of cases where test system 2 did not alarm (11 cases) but actually 
contained SNM cargo (1 case) 1/1 1 = 0.09 

C'learly, test system 1 appears to be less reliable than test system 2. The calculation of the 
lower bounds on these estimated probabilities is not as straightforward as using the binomial 
distribution, as was done for sensitivity and specificity, because the denominator (10 in the 
outcome of the performance tests of system 1 and 1 1 in the outcome of the performance tests on 
system 2) arose from the test results, not from the number of trials set by the study design. That 
is. the denominator “10” for test system 1 (and “11” for test system 2) is the sum of two numbers 
that might differ if the test were re-run. Confidence bounds can be obtained as a function of 
sensitivity (S) and specificity (T) (see Note 2). 

In formal notation, we estimate the probability of a false negative call from estimates of 
sensitivities and specificities, we use the following notation. Let A and B denote two events, say 

A = cargo contains SNM 

B = Test system alarms 

A c = The complement of A, cargo contains no SNM (benign) 

B c = The complement of B. test system does not alarm 

The FNCP is the probability that event A occurs (cargo truly contains SNM). given that 
event If occurred (test system does not alarm). We write this probability as ?{A\Bf}. (The event 
after the vertical bar is the event on which the probability is conditioned, i.e., the event that 
exists.) 

Bayes’ rule(Navidi, 2006) states: 
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P{A\B c }=P{B ! \A)xP{A)/[(P{B c |/l}x/ 3 {/l})+(P{S c |/f}xP{/T})] 

0) 

where 

= probability that event A occurs, given confirmation that event B has occurred 
(here, P{cargo contains SNM | test system does not alarm} 1 NPV) 

Pfi^l/l } = probability that event B c occurs, given confirmation that event A has occurred 
(here, P{test system does not alarm | cargo contains SNM} = 1 S) 

P{£%4 C } ~ probability that event B occurs, given confinnation tliat event A c has occurred 
(here. P{test system does not alarm | cargo contains no SNM } = T). 

Recall that sensitivity is the probability that the test system alarms, given SNM was in the 
cargo: i.e., P{5[/1} = sensitivity (S). Both S and T can be estimated from the experimental test 
runs (where we know what the cargo contained). Denoting by p x the probability that cargo 
contains SNM, we have: 


FNC, "ira^)|-S7' <2 > 

where y = [77(1 -S)] < [(1 -p)/p]. We prefer systems w ith low er values of this probability; i.e., with 
higher values of y. 

Denoting by 5,, 7], S 2 . T 2 the sensitivities and specificities of systems 1 and 2, 
respectively, we prefer system 1 to system 2 if FNCPj < FNCP: ; i.e.. if 


yi > y2 


i.e., if 


f T < ) 

f 1_/ 0 


' T 2 ' 

(I-P 

li-sj 

l p ) 


[l-Sj 

{ P 


which is the same as either 



T, . 1-S, 
t 2 1 -S,' 


( 3 ) 

( 4 ) 


Tliat is. a comparison of FNCP for test system 1 (FNCPj) with that for test system 2 
(FNCP:) reduces to a comparison of [( 1 -sensitivity) (specificity )1 for the two systems. We can 
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estimate uncertainties on our estimates of sensitivity and specificity (based on the binomial 
distribution: see above discussion). Hence, we can approximate the uncertainty in [(1 - S^T)], 
and ultimately the uncertainty in the ratio of false negative call probabilities (see Note 2) — 
which does not involve assumptions on p (likelihood of the threat). Notice that test system 1 is 
always preferred if Tj > T 2 and Sj > S>. because Tj > T: implies that the left-hand side of (4) 
exceeds or equals 1. and Sj > S » implies that the right-hand side of (4) is less than or equal to 1; 
hence (4) is satisfied. (If 7/ = T> and Si = S 2 , then the test systems are equivalent, in terms of 
sensitivity, specificity, and false negative call probability, so either can be selected.) In real 
situations, however, one test system may have a higher test sensitivity may but a lower 
specificity. For example, if 7 1 , 0.70 and T 2 = 0.80 (test system 2 is more likely to remain silent 

on truly benign cargo than test system 1). but 0.950 and S, 0.930 (test system 1 is slightly 
more likely to alarm if the cargo truly contains SNM), then (4) says that test system 1 is 
preferred, because TJT 2 0.875 and (US’ )/(l-S 2 ) » 0.05/0.07 0.714. Hie FNCP for the two 

systems are 


j — 

FNCP, =■ 

1 + | 

' ) 
2-s,) 

l 

(^1 


'll 14 00 (»-p)1 
P 

1 

1 + 

/ rp > 

1 1 



n 11.43(l-p)‘ 
P 


so clearly FNCP<FNCP r 

Calculations for this example (S, 0.95, S 2 = 0.93, T l 0.70. 7\ - 0.80). for different 

threat levels p, arc: 

• p = 0. 10: FNCP { = 0.007874 and FNCP 2 ~ 0.009629 (ratio - 0.81777); 

• p = 0.05: FNCP { = 0.003745 and FNCP 2 - 0.004584 (ratio = 0.81701); 

• p = 0.01: FNCP y =0.000721 and FNCP 2 = 0.000883 (ratio = 0.81646); 

• p = 0.001 : FNCP t = 0.715010-’ and FNCP^ = 0.8758’ lO ’ (ratio = 0.81634); 

• p = 0.0001 :FNCP I - 0.7142 10’ 5 and FNCP, = 0.8751 10"' (ratio = 0.81633). 

Hie prevalence p has little effect on the ratio of FNCPs, but its effect on the absolute rate 

(magnitude) of the FNCP is noticeable. Regardless of its value, however, the probability of a 
FNC will be very small whenever the probability of a threat is small (e.g.. less than 0.1). 

When the differences in sensitivities are much higher, the FNCPs also are quite different. 
Consider the case when = 0.90, S 2 = 0.30, 7^ = 0.70, T 2 = 0.90, for the same threat levels: 

• p = 0. 10: FNCP X = 0.01 5625 and FNCP 2 = 0.079545 (ratio = 0. 19643); 

• p = 0.05: FNCP X = 0.007463 and FNCP 2 = 0.038326 (ratio = 0. 18977): 

• p = 0.01: FNCP X =0.001441 and FNCP 2 = 0.007795 (ratio = 0.18485); 

• p = 0.00 1 : FNCP X = 0.000 1 43 and FNCP 2 = 0.000780 (ratio = 0. 1 8379); 
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• p = 0.000 1 : FNCP l = 0.1429 1 0~* and FNCP 2 = 0.7778 ' 1 O' 4 (ratio = 0. 1 8369). 

Here, even with a higher specificity, the increase in sensitivity from 0.3 (test 2) to 0.9 
(test 1) results in a five-fold decrease in the FNCP. With either test, the FNCP is small, even 
when the threat level is 0.01 (1 in 100 trucks earn' threatening cargo). 

Calculations for the probability of a false positive call (FPCP. 1-PPV) are similar. Again 
from Bayes* Theorem: 

P{A\ff) = p{B | A)xP{A}/l(P{B ! M}x/>{.4})+(/>{fl c (5) 

where 

A c complement of A event that cargo does not contain SNM 
I? = complement of B = event that test system does not alarm 

?{A C \B) = probability that event A c occurs even though B occurred(here. P{cargo 
contains no SNM ! test system alarms) - 1 PPV) 

P{2?%4 C } = probability that event B occurs, given confirmation that event A c has occurred 
(here. P{test system does not alarm | cargo contains no SNM} - T). 

P{£%4} = probability that event Bf occurs, given confirmation that event A has occurred 
(here. P{test system does not alarm | cargo contaias SNM} = 1 S) 

FPCP = (irxip)/|(irxi^)^| - 1/(1+*) where z = [S/(\T))Qlp/(\p)\. 

So test system 1 would be preferred, in these terms, over system 2, if 



To calculate the magnitude of FPCP (not just the ratio of the probabilities for the two 
systems), consider that p is likely small and that Sj (orS 2 ) may not be orders of magnitude large r 
than (l-Tj) (or (l-7* 2 ). In this case, the “1 +” in the denominator does matter for the absolute 
magnitude of this FPCP. For the example above, where S } 0.95. S 2 = 0.93, T, = 0.70. T 2 - 0.80. 
the corresponding FPCP forp=0.10.p= 0.05, p = 0.01. p - 0.001. p = 0.0001 are: 

• p = 0.10: FPCP, = 1/[1 + 0.31579(1/9)J = 0.96610, FPCP 2 = 0.97666 (ratio = 
0.9892) 

• p = 0.05: FPCP, = 0.98365 . FPCP 2 = 0.9888 1 (ratio = 0.99478) 

• p = 0.01: FPCP, = 0.99682 , FPCP 2 = 0.99783 (ratio - 0.99899) 

• p = 0.00 1 : FPCP, = 0.99968. FPCP Z = 0.99978 (ratio = 0.99990) 

• p = 0.0001 : FPCP, = 0.99997. FPCP 2 = 0.99998 (ratio = 0.99999). 

For these examples, the chance of having to re-inspect every sounded alarm, only to find 
benign material, is virtually identical in both systems (and very close to 1 for both). The same is 
true when S, =0.90. S 2 =0.30, T, =0.60. T 2 =0.80: 
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• p - 0.01: FPCP X =0.95294. FPCP 2 =0.93103 (ratio - 1.02353) 

• p = 0.05: FPCP X =0.97714, FPCP t = 0.96610 (ratio = 1.01143) 

• p = 0.01: FPCP X =0.99553, FPCP 2 = 0.99331 (ratio 1.00223) 

• p - 0.001: FPCP X =0.99956, FPCP 2 =0.99933 (ratio - 1.00022) 

• p = 0.0001: FPCP X = 0.99996, FPCP Z = 0.99993 (ratio = 1.00002). 

Hie DNDO criteria for “significant improvement in operational effectiveness” involve 
comparisons of sensitivity and specificity. As noted above, a test system that has higher 
sensitivity and higher specificity will have a lower false negative rate. But the above calculations 
also demonstrate that “nearly equal" sensitivities and specificities result in nearly equivalent 
systems, and hence offer rather limited benefit for the cost. For completeness, we re-write the 
DNIX) criteria for “significant improvement in operational testing" (see Box 2. pp 40 41) using 
the S, T notation (for sensitivity and specificity). 

Let Sj ‘ (SNM , noNORA / ) denote the sensitivity of the ASP system in primary (1) 
screening when the cargo truly contains SNM and no NORM; i.e.. S^’(SNM , noNORM ) = 
P{ASP alarms | cargo contains SNM, no NORM}. Likewise, let S' ” ’ (SNM , noNORM ) denote 
the sensitivity of the current (PVT+RIID) system in primary (1) screening when the cargo truly 
contains SNM and no NORM; i.e., S”’ (SNM , noNORM ) - P{PVT alarms in primary screening | 
cargo contains SNM. no NORM} Using T to denote specificity, let T, *(SNM , noNORM) = 
P{ PVT RIID does not alarm in secondary screening | cargo contains no SNM, but possibly 
NORM } (specificity). 

Denote by sj * and the sensitivities of ASP and PVT+RIID combination, respectively, 
in primary screening, and T*a and 7/ the specificities of ASP and PVT+RIID, respectively; 
superscript (2) indicates secondary screening. DNDO has specified its criteria for “operational 
effectiveness" as follows: 

1. S ( "(SNM , noNORM) a S™(SNM, noNORM) 

2. S"'(SMf + NORM) 2 : S ( "(SNM + NORM) (different version of criterion 1 
above) 

3. T[ U (MI - Iso) 2. T™ (MI - Iso) (where “Ml-Iso" indicates “licensable medical or 
industrial isotopes). 

4. 1 - T™(NORM) < 0.20(1 - T^(NORM)] 

=> 0. 8 < T (NORM ) - 0.2(7/ > (NORM )) . 

5. 1 - Sf(SNM) <> 0.5S™(SNM ) => 0.5 £ S'" (NORM )-0.5(S" ] (NORM)) . 

6. Time in secondary for ASP < time in secondary for RIID (no connection to 
sensitivity/specificity). 

Since criterion 4 is more stringent than criterion 3 and criterion 5 is more stringent than 
criterion 1. we concentrate on values of sensitivity and specificity that satisfy criteria 4 and 5. 
When these two conditions are satisfied (i.e., Ta > 0.8 + 0.27V and Sa > 0.5 + 0.5S>). the ratio of 
false negative call probabilities (A to B) can be as small as 1:900 almost 1000 times smaller. 
For such improvements, the ratio of both the sensitivities and the specificities must be on the 
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order of 0.99/0.10 or 0.95/0.10; in such cases, the false negative call probabilities are on the 
order of (10 3 to 10' 5 ). Tables of values of the probabilities of both false negative calls and false 

positive calls were calculated when T A , S A , T f , and S r were set equal to 0.1, 0.2 0.8, 0.9, 

0.95, 0.99; of the ll 4 = 14,641 combinations, only 858 satisfied criteria 4 and 5. These 858 
combinations were set along with 5 different values of p = 0.01 (cargo is present in 1 of 100 
trucks), 0.001, 0.0001, 0.00001, 0.000001 (1 in 1,000,000 trucks). A plot of the smaller false 
negative call probability (denoted FNCPj in the figure) versus the larger one (denoted FNCPi) is 
shown in Figure B.l. (the red dashed line corresponds to the line where the two false negative 
call probabilies are equal). The upper left comer shows the cases where the FNCPs are most 
different (0.00112 <FNCPJFNCP ] <0.00311), which occurred in 26 of the 858 cases (26D5 
points are shown, corresponding to 5 values of p). More frequently, the ratio is less dramatic 
(0.00317 <FNCPJFNCP 2 <0.03161 for 257 of the 858 cases; 
0.0316 <FNCPJFNCP 2 <0.3162 for 535 of the 858 cases; 0.3165 < FNCPJ FNCPi <0.4819 
for 40 of the 858 cases). In each case, the absolute magnitudes of the false negative call 
probabilities are quite small, and the ratios of the false positive call probabilities are almost 1. 
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Figure B.l: Plot of FNCP 2 versus FNCP { for cases satisfying the criteria T A >0.8+0.27), and 
S A >0.5 + 0.55, , for different levels ofp (1 x 10 J , 1 x 10' 3 , lx 10 4 , 1 x 10 5 , lx 10' 6 ). Tliered 
dashed line corresponds to TWC/J = FNCP 2 . The results are stratified by magnitude of the ratio 
FNCPJ PNC (specifically, rounded values of the common logarithm of the ratio: -3, -2, -1, 
0, respectively, for the four plots). 
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Note 1 : A comment on notation 

We denoted by FNCP the probability of making a false positive call and by FPCP the 
probability of making a false positive call; i.e., 

FNCP = P{ true + 1 test calls } 

FPCP = P{ true - 1 test calls “+" } . 

We related these probabilities to the following generic two-way tabic of test outcomes (notation 
from Benjamini and Hochberg 1995, p.291, is in parentheses): 

Test calls Test calls Total 

Truth “Positive” “Negative” Tests 

True POSITIVE A r ^.(l / ) N, = m - m 0 

True NEGATIVE (S) N_ (7) N. = m 0 

Total calls R m-R m 


We estimated the false negative call probability via the proportion of negative-call tests (mR) that 
were in fact positive (/V 4 ), or U/(m-R ) in BH95 notation. Similarly, we estimated the false 
positive call probability via the proportion of positive-call tests ( R ) that were in fact negative 
(A/ 4 ). or V/R in BII95 notation. BII95 address the situation known as “multiple testing.” where 
one is conducting many hypothesis tests (e.g.. hundreds or thousands of tests as occurs in gene 
expression experiments), and wants to control the frequency with which one declares as 
“significant” (e.g.. “positive”) tests which in fact are negative. Hence Benjamini and Hochberg 
(1995) define the expected proportion of false positive calls. E(V/R), as the “false discovery 
rate.” or FDR. They provide a procedure based on the m p-values from the m tests so that one has 
assurance that, on average, the proportion of "declared significant" tests that in fact are not 
significant remains below a pre-set threshold (e.g.. 0.05). If we estimate the FPCP as V/R, we can 
think of this estimated FPCP as an estimate of Benjamini and Ilochberg's FDR. In analogy with 
E(V/R)~FDR, some have termed E(U/(mR)) the “false non-discovery rate.” 

Our situation differs from the multiple testing situation in two ways. First, our two-way table 
arises from a designed experiment where values of m 0 and m lire set by design. Second, our 
bigger concern lies not with false positive calls but rather w ith false negative calls; i.e.. with the 
probability that a cargo declared “safe” (negative) actually is dangerous (true positive). The table 
suggests that we can estimate FNCP as U/(mR). Some authors have called the expected value of 
this ratio. E(U/(mR)). the “false non-discovery rate” (see Genovese and Wasserman 2004: Sarkar 
2006). But with both FNCP and FPCP. one needs further information about the frequency of true 
“positives” and true “negatives” (in the form of p ~ probability that cargo contains SNM or other 
threatening material) beyond the m tests given in the design. In fact, as further tests are 
conducted, better estimates of FNCP and FPCP can be obtained by incorporating better estimates 
of sensitivity and specificity, as well as p. into the formulas for FNCP and FPCP. For that reason, 
we have chosen to derive the relevant probabilities using Bayes* formula rather than using the 
terms “false discovery rate” and “false non-discovery rate.” which often are estimated from only 
the table of outcomes from multiple tests. For further information, see the references below. 
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67 


Note 2: Uncertainty m the ratio FNCP j /FNCP 2 

The uncertainty in the ratio FNCP j FNCP 2 » (( 1 -S j )/ Tj ] /[ ( 1 -.S^ V ^ | (( 1-Sj yD-Sf^WTjJT j ] can 

be approximated using propagation of error formulas. Let ratio = N/D denote a generic ratio (N 
= Numerator, D ~ Denominator). 

SE(ralio) = SE(N/D ) * ratio x l,r <; v > + 

V N 1 D 1 


When T and S have binomial distributions, Var{T X )=T X (\-T x )!n v Var{ S ] y=S { ( 1 -S { )!n { and likewise 
for Var(T 2 ) and l'ar(S 2 ), where n x (hJ is the number of trials on which S x and 7 J [S', and 7,1 are 
estimated (in experimental runs at Nevada Test Site, n^n^t 12 or 24). Hence, the standard error 
(square root of the variance) of (1 -Sj)/T] is approximately 


^ His, 


» - 7 -, 

nj, 


so the standard error of Uie ratio of false negative call probabilities (when p is tiny) is 
approximately 


a/ FNCP ' ) ~ f FNCP ' If 


Vatjf-NCP, ) I'aijFNCP,) 
FNCP, 1 FNCP‘ 


So. 


SE(FNCP, IFCNP, )* 
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Appendix C 


The Value of Factorial Experiments 

Factorial experiments are extremely useful designs when outcomes are needed for a 
variety of test conditions. For example, consider the following factors that could affect test 
performance (e.g., probability of an alarm, or probability of no alarm): 


• Masking (absent present) 

• Shielding (absent, present) 

• Mask location (front, middle) 

• Mask height (front, middle) 

• Shield location (front, middle) 

• Shield height (front, middle) 

• SNM (none, some) 

• NORM (none, some) 


More than 8 factors could be envisioned (e.g., cargo density, ambient temperature, 
ambient humidity, background radiation level), and more than just 2 levels for each factor could 
be considered. For example, the masking and shielding factors could have levels labeled 
“absent.” “front.” and ‘middle.” and the SNM and NORM factors could have four levels labeled 
“none,” “small,” “medium.” and “large,” resulting in a 3x3x4x4 design (a total of 144 test 
conditions). This appendix illustrates the value of factorial designs (and a way to reduce the 
number of test conditions) with the above design simply for ease of illustration. The same 
concepts apply to more complex designs. But even with only these 8 factors at these levels, the 
testing of all 2 x 8 - 16 single-factor tests would not be informative. For example, what happens 
if a cargo contains some SNM and some NORM with much shielding and some masking placed 
in the middle of the truck? None of the 16 test runs would answ er this question. One might also 
want to know if the probability of detecting SNM is affected by the combined presence of 
masking and shielding of different magnitudes a question that likew ise would not be answered 
by any of the 16 runs. 

The benefits of running test combinations can be seen already with the follow ing 
(simpler) test design with these hypothetical results: 


masking 


shielding 

present absent 

present 0.20 0.95 

absent 0.80 0.99 


If one tested only “masking present” and “masking absent” in the absence of shielding, one 
might conclude that masking has some effect on SNM detection (0.95 vs. 0.99), but not as great as 
the effect of shielding in the absence of masking (0.80 vs. 0.99). One needed 3 runs to ascertain 
this conclusion. But with only one more run (masking and shielding both present), one sees that 
their combined effect is devastating to the probability of detection (0.20) — far lower than w ith 


69 

Prepublication Copy 


Copyright © National Academy <A Sciences- All rights reserved. 


VerDate Oct 09 2002 09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00174 Fmt 6601 Sfmt 6602 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



169 


70 


EVALUATING TESTING. COSTS. & BENEFITS OF ASPs: INTERIM REPORT 


either factor singly. The effect of different combinations of factors can be especially illuminating; 
hence the value of experimental designs with combinations of factors or “factorial designs.” 

Unfortunately, testing all 2x2x2x2x2x2x2x2 ~ 2 8 ~ 256 combinations would be 
infeasible, especially since the outcome of each test is a ‘ probability of detection”; i.e.. (number 
of runs that sounded alarm) (total number of runs). To minimize the uncertainty in this estimated 
probability, several runs must be conducted at each test scenario. With only n=6 or n=12 runs, 
one would have to conduct 256x6 = 1536 or 256x12 - 3072 test mns, and. even then, the 
uncertainty in the estimated probability could be as high as 30%-40% (95% confidence). For 
example, a perfect test of 6 correct actions (6/6) would yield an approximate 95% confidence 
interval for the true probability of detection as [(1-0.95) 1 * 1 , 1| = (0.61, 1.00) if n 6 or [(1- 
0.95) l/h , 1] (0.78, 1.00) if n = 12. Clearly some reduction in the number of test scenarios is 

needed. 

Fractional factorial experiments are factorial experiments w ith only a fraction of the total 
number of runs. Consider, for ease of illustration, only 4 factors, denoted A, B. C, D. each at 2 
levels (“present”, “absent”). Sixteen test scenarios would cover all combinations, as follows: 




Factor levels 



Product (Mod 2) 

Scenario 

A 

B 

c 

D 

ABCD 

1 

1 

1 

1 

1 

1 

2 

1 

i 

1 

0 

0 

3 

1 

1 

0 

1 

0 

4 

1 

1 

0 

0 

1 

5 

1 

0 

1 

1 

0 

6 

1 

0 

1 

0 

1 

7 

1 

0 

0 

1 

1 

8 

1 

0 

0 

0 

0 

9 

0 

1 

1 

1 

0 

10 

0 

1 

1 

0 

1 

11 

0 

1 

0 

1 

1 

12 

0 

1 

0 

0 

0 

13 

0 

0 

1 

1 

1 

14 

0 

0 

1 

0 

0 

15 

0 

0 

0 

1 

0 

16 

0 

0 

0 

0 

1 

“1” - “present”, “0” 

= “absent"; 

“Product (Mod 2)” = 

1 with even numbers of 1's, 0 with odd numbers of 1 

Consider the rows whose last column value is 1 : 




Run if 

A 

B 

c 

D 


1 

1 

1 

1 

1 


4 

1 

1 

0 

0 


6 

1 

0 

1 

0 


7 

1 

0 

0 

1 


10 

0 

1 

1 

0 


11 

0 

i 

0 

1 


13 

0 

0 

1 

1 


16 

0 

0 

0 

0 
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Notice that exactly 4 runs have A absent and 4 runs have A present; the same is true of B, 
C, or D. Moreover, when A is present (first 4 runs), exactly 2 of the 4 runs have B present and 2 
have B absent; the same is true for C and D. and any two of the four factors (A and C, A and D, 
etc.). In fact, all 8 runs for any combination of 3 factors (A, B. C: A, B, D; B. C. D) are included. 
So this design allows us to evaluate: 

• The effect of A (present vs. absent) 

• The effect of B 

• The effect of C 

• The effect of D 

• The effect of A and B together 

• The effect of A and C together 

• The effect of A and D together 

• The effect of B and C together 

• The effect of B and D together 

• The effect of C and D together 

• The effect of A. B. and C together 

• The effect of A. B. and D together 

• The effect of B. C, and D together 

The only effect that we cannot assess is the 4-way interaction, ABCD. But we have 
reduced the number of runs from 16 to 8, a big savings. 

Hie same principle applies with 8 factors. If resources allow as to run only 64 scenarios, 
then we sacrifice the ability to estimate the interactions that involve 5 or more factors at once — 

e.g., ABCDEFGH. all 7-factor interactions (ABCDEFG BCDEFGH) — but we can estimate 

all other main effects and 2-way. 3-way. and 4-way interactions. (Usually interactions involving 
4 or more factors are hard to interpret anyway.) If we can run only 32 scenarios, we sacrifice the 
ability to estimate not only these high-order interactions, but also some ability to resolve some 
two-factor interactions; but we can still assess the main effects (A alone. .... H alone) and most 
two-factor interactions (AB, .... GII) — all with just 32 runs, a huge savings. 

The designs that NIST provided to DNDO for their test runs followed this principle. The 
only limiting factors are n. the number of test runs, and the inability to conduct the “SNM 
present*’ tests as blind tests. The former can be improved by increasing n; the latter can be 
addressed by hiring “actors’* to pretend to act as security agents, with only DNDO personnel 
aware of the true SNM test scenarios. The effect of bias when tests are run unblinded has been 
documented extensively in the medical literature; unblinded tests must be viewed with great 
caution and even skepticism. 
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Appendix D 

Brief Biographies of Committee Members 

Robert C. Dynes, chairman of the committee, is a professor of physics at the San Diego and 
Berkeley campuses of the University of California, where he directs laboratories that focus on 
superconductivity. From 2003 until 2008, he served as the 18th president of the University of 
California (UC) and before that as chancellor of UC San Diego. As a professor, he founded an 
interdisciplinary' laboratory in which chemists, electrical engineers, and private industry 
researchers investigated the properties of metals, semiconductors, and superconductors. Prior to 
joining the UC faculty, he had a 22-vear career at AT&T Bell Laboratories, where he served as 
department head of semiconductor and material physics research and director of chemical 
physics research. Dr. Dynes received the 1990 Fritz London Award in Low Temperature 
Physics, was elected to the National Academy of Sciences in 1989. and is a fellow of the 
.American Physical Society, the Canadian Institute for Advanced Research, and the .American 
Academy of Arts and Sciences. He serves on the Executive Committee of the U.S. Council on 
Competitiveness. A native of London, Ontario, Canada, and a naturalized U.S. citizen. Dr. Dynes 
holds a bachelor's degree in mathematics and physics and an honorary doctor of laws degree 
from the University of Western Ontario, and master's and doctorate degrees in physics and an 
honorary doctor of science degree from McMaster University. He also holds an honorary 
doctorate from L’Universite de Montreal. 

Richard E. Bluliut is the Henry Magnuski Professor of Electrical and Computer Engineering at 
the University of Illinois and the head of that department. He also holds the title of research 
professor in the Coordinated Science Laboratory. From 1964 to 1994. Blahut was employed in 
the Federal Systems Division of IBM. where he had general responsibility for the analysis and 
design of coherent signal processing systems, digital communications systems, and statistical 
information processing systems. He was responsible for the original development of passive 
coherent location systems, now a major technique used in the U.S. Department of Defense. Other 
contributions to industry include the development of error-control codes for the transmission of 
messages to the Tomahawk missile, codes to protect text data transmitted via the U.S. public 
broadcasting network, and the design of a damage-resistant bar code for the British Royal Mail. 
Dr. Blahut has authored a series of advanced textbooks and monographs in error-control coding, 
information theory, and signal processing, including ten books either published or in manuscript 
form. Dr. Blahut served as president of the Information Theory Society of the Institute of 
Electrical and Electronics Engineers (IEEE) in 1982, and was editor-in-chief of the IEEE 
Transactions on Information Theory from 1992 until 1995. He was elected to the National 
Academy of Engineering in 1990. He is a fellow of the IEEE. He is the recipient of the IEEE 
Alexander Graham Bell Medal, the IEEE Claude E. Shannon Award, the Tau Beta Pi Daniel C. 
Drucker Eminent Faculty Award, and an IEEE Millennium Medal. He received his Ph.D. degree 
in electrical engineering from Cornell University. 

Robert R. Borchers. a physicist and expert in computation, is chief technology officer for the 
Maui High Performance Computing Center at the L’niversitv of Hawaii. Prior to joining the 
Lhiversity of Hawaii, he served eight years at the National Science Foundation as director of the 
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Division of Advanced Computational Infrastructure and Research. Earlier in his career, he was a 
professor of physics before holding several high-level management positions in universities and 
laboratories, including associate director for computation at Lawrence Livermore National 
Laboratory, vice chancellor for academic affairs at the University of Colorado Boulder and the 
University of Wisconsin, Madison, and director of the Physical Sciences Laboratory at Madison. 
Dr. Borchers has received numerous awards and is a fellow of the American Physical Society. 
Dr. Borchers received his B.S. degree from the University of Notre Dame and M.S. and Ph.D. 
degrees from the University of Wisconsin. Madison, all in physics. 

Philip E. Coyle III served as assistant secretary of defense and director, operational test and 
evaluation, in the Department of Defense (DoD). In this capacity, he was the principal advisor to 
the secretary of defense and the under secretary of defense for acquisition, technology and 
logistics on test and evaluation in the DoD. Mr. Coyle has 30 years experience in testing and 
test-related matters. From 1959 to 1979. and again from 1981 to 1993. Mr. Coyle worked at 
Lawrence Livermore National Laboratory where he served as an associate director of the 
Laboratory. During the Carter Administration. Mr. Coyle served as principal deputy assistant 
secretary for defense programs in the Department of Energy. In this capacity he had oversight 
responsibility for the nuclear weapons testing programs of the Department. Hie International 
Test and Evaluation Association awarded Mr. Coyle the Allan R. Matthews Award, its highest 
award, for his contributions to the management and technology of test and evaluation. Mr. Coyle 
was awarded the Defense Distinguished Service Medal by Defense Secretary William Perry, and 
the Bronze Palm of the Defense Distinguished Service Medal by Defense Secretary William 
Cohen. Mr. Coyle graduated from Dartmouth College with a B.A. degree and an M.S. degree in 
mechanical engineering. 

Roger L. Hagengruber is the director of the Office for Policy. Security and Technology 
(OPS&T) and the Institute for Public Policy (IPP) and a research professor (political science and 
physics) at the University of New Mexico. Previously, he served as chief security officer and 
chief cyber security officer for Los Alamos National Laboratory and as a senior vice president at 
Sandia National Laboratories and directed Sandia’s primary mission in nuclear weapons during 
the transition following the end of the Cold War. Dr. Hagengruber spent much of his 30-vear 
career at Sandia in arms control and non-proliferation activities including several tours in 
Geneva as a negotiator. In recent years, he has focused on the nuclear transition in the former 
Soviet Union and on security issues associated with counter-terrorism and has chaired or served 
on numerous panels that have addressed these issues. His work at the University of New Mexico 
includes directing the IPP work in public surveys including sampling of U.S. and European 
views on a wide range of security issues. The OPS&T creates multidisciplinary teams from labs 
and universities to explore policy options for issues in which security and technology are 
interrelated. He previously served on the Nuclear and Radiological Panel of the National 
Research Council's Committee on Science and Technology for Countering Terrorism. He 
received his Ph.D. degree in experimental nuclear physics from the University of Wisconsin and 
is a graduate of the Industrial College of the Armed Forces. 

Carl N. Henry retired from Los Alamos National Laboratory in December 2001. where he 
worked for over 40 years. Following retirement, he did part-time consulting for Sandia National 
Laboratories at LIS. Department of Energy headquarters in 2006. From 1994 to 2001. Henry 
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worked on foreign nuclear weapons intelligence and counter-intelligence analysis. In 1997, he 
received the Intelligence Community Seal Medallion for meritorious service. From 1975 until 
1994, he worked on the Nuclear Emergency Search Team (NEST) Program. During that time he 
served many roles including staff member, group leader, and program manager. Over his career 
Henry has participated in search and diagnostics activities, real deployments and exercises, and 
led the planning for one major exercise. In addition, he has conducted nuclear safeguards 
research as part of a team using active analysis of nuclear material with a Cockcroft-Walton 
accelerator and neutron and ray detectors for portal monitoring applications. 

John M. Holmes is deputy executive director of operations at the Port of Los Angeles, 
overseeing the operations of the Los Angeles Port Police, the Homeland Security Division, 
emergency preparedness planning, the construction and maintenance department, and the Port 
Pilot Service. Before his current position, he most recently served as a principal and chief 
operating officer of the Marsee Group, a full service security consulting firm specializing in 
supply chain security, technology and operations. Prior to forming the Marsee Group. Captain 
Holmes held the position of vice president and director of business development for Science 
Applications International Corporation (SAIC), where he assisted government and commercial 
customers in the development of technological solutions to homeland security challenges, with 
emphasis on port, border and military security solutions. Captain Holmes retired from the United 
States Coast Guard in 2003 with 27 years of service as commanding officer, officer in charge of 
marine inspection and captain of the Port for the Los Angeles-Long Beach port complex. While 
in the Coast Guard, he also served as deputy chief of the Coast Guard Office of Congressional 
Affairs, was attached to the staff of the governor of American Samoa and the U.S. ambassador to 
the Republic of Singapore, and also served as delegate and committee chairman at the 
International Maritime Organization in London. Captain Holmes received bachelor's degrees in 
English and education from Boston College, and an M.B.A. degree from the John M. Olin 
School of Business at Washington University in St. Louis. 

Kami Kafadar is the Rudy Professor of Statistics in the College of Arts and Sciences. Indiana 
University, Bloomington. Her research focuses on robust methods, data analysis, and 
characterization of uncertainty in the physical, chemical, biological, and engineering sciences. 
Prior to joining the Indiana faculty in 2007, she was chancellor's scholar and professor of 
statistics and director of the Statistical Consulting Service at the University of Colorado. Denver. 
Earlier appointments include National Institute of Standards and Technology (NIST). Hewlett 
Packard, and the National Cancer Institute. She is currently serving as chair of the NRC’s 
Committee on Applied and Theoretical Statistics (CATS) and on the Board of Mathematical 
Sciences and their Applications (BMSA). She has served as Editor or Associate Editor on several 
editorial review boards and on the governing boards of the American Statistical Association 
(ASA), the Institute of Mathematical Statistics, and the International Statistical Institute (ISI). 
Dr. Kafadar is a fellow of the ASA and the ISI. and has authored over 80 journal articles and 
book chapters. She received her B.S. in mathematics and M.S. in statistics from Stanford in 1975 
and her Ph.D. in statistics from Princeton in 1979. 

C. (C harles) Michael Lederer is a research chemist and deputy director emeritus of the 
University of California Energy Institute, where he is responsible for the planning and 
management of the Energy Institute's grant programs. For 20 years, he was a lecturer teaching 
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courses in radiation detection and measurement, and chemical methods in nuclear technology in 
the Department of Chemistry and the Department of Nuclear Engineering at the University of 
California at Berkeley. Prior to joining the Energy Institute, Dr. Lederer was head of the 
Information and Data .Analysis Department and director of the Isotopes Project at Lawrence 
Berkeley Laboratory. He is most widely known as co-author of the 6th and 7th editions of the 
Table of Isotopes, for which he evaluated nuclear structure and decay data for all known nuclides 
and computerized the Isotopes Project. Dr. Lederer received an A.B. degree in chemistry from 
Harv ard University and a Ph.D. degree in nuclear chemistry from the University of California at 
Berkeley. 
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HEARING CHARTER 

SUBCOMMITTEE ON INVESTIGATIONS AND OVERSIGHT 
COMMITTEE ON SCIENCE AND TECHNOLOGY 
U.S. HOUSE OF REPRESENTATIVES 

The Science of Security, Part II: 
Technical Problems Continue to 
Hinder Advanced Radiation Monitors 

TUESDAY, NOVEMBER 17, 2009 
i:oo p.m.-3:oo p.m. 

2318 RAYBURN HOUSE OFFICE BUILDING 


Purpose 

The Subcommittee on Investigations and Oversight meets on November 17, 2009, 
to examine continuing problems with the Department of Homeland Security’s (DHS) 
efforts to acquire its next generation radiation monitors known as Advanced 
Spectroscopic Portals (ASPs). This is a follow-up to the hearing the Subcommittee 
held on June 25, 2009, titled: The Science of Security: Lessons Learned, in Devel- 
oping, Testing and Operating Advanced Radiation Monitors. Since the Domestic Nu- 
clear Detection Office (DNDO), a DHS component, was created in 2005 they have 
been responsible for researching, developing, testing and managing the program. 

The ASP program is estimated to cost $2-to-$3 billion and has been under scru- 
tiny since 2006 for failing to have clear-cut requirements, an adequate test plan, 
sufficient timelines, development milestones or a transparent and comprehensive 
cost benefit analysis. These problems have been identified by the Government Ac- 
countability Office, National Academy of Sciences, the Homeland Security Institute, 
a Federally Funded Research and Development Center for DHS, and the National 
Institute of Standards and Technology. 

In July, one month after the Subcommittee’s last hearing, the ASPs went through 
a second round of Field Validation Tests. During the tests the ASPs exhibited sev- 
eral “false positive” alarms for special nuclear material that did not exist. In an- 
other disturbing incident during the tests, one ASP monitor stopped working alto- 
gether yet the system operator remained unaware of this malfunction. Two dozen 
cargo trucks were permitted to go through the non-functioning portal monitor in 
order to be screened for potential radioactive and nuclear material until the problem 
became apparent. DNDO considered this a “Mission Critical Failure.” No new plans 
have yet been scheduled to re-test the ASPs for the third time. The Subcommittee 
will examine the results from the most recent tests, continuing technical problems 
with the ASPs, supply shortages of a key component for radiation monitors that 
may hinder the eventual deployment of the ASPs and further drive up its potential 
cost, and potential enhancements to the current fleet of radiation monitors in use 
today. 

Background 

Since the September 11, 2001 terrorist attacks, protecting the Nation from a nu- 
clear or radiological attack has been a top national security priority. In 2002, to help 
address this threat, the U.S. Customs and Border Protection (CBP) agency began 
deploying radiation monitors at U.S. border sites and ports of entry to screen the 
more than 23 million cargo containers that enter the country every year for radio- 
logical and nuclear materials. 

Polyvinyl toluene (PVT) radiation portal monitors have been used to screen this 
cargo since then. They are able to detect the presence of radioactive sources, but un- 
able to identify the type of radiation present. The PVT monitors, while relatively 
inexpensive, robust and highly reliable, are unable to distinguish between radio- 
active sources that might be used to construct a nuclear bomb, such as Highly En- 
riched Uranium (HEU), and non-threatening naturally occurring radiological mate- 
rials (NORM) contained in ceramic tiles, zirconium sand or kitty liter, for instance. 
As a result, any time a PVT detects a radioactive source the cargo is sent to “sec- 
ondary” screening where CBP agents verify the detection of the source with a sec- 
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ond PVT monitor and use hand-held Radioactive Isotope Identification Devices 
called RIIDs to help identify the source of radiation. 

This method of operation leads to many “secondary” inspections for naturally oc- 
curring radioactive material or radioactive material intended for benign purposes, 
such as radioactive medical isotopes. At the Los Angeles/Long Beach port of entry, 
for instance, PVT monitors routinely send up to 600 conveyances of cargo to sec- 
ondary inspection each day. In addition, the RIIDs used in secondary inspections are 
limited in their abilities to locate and identify potential radioactive material in large 
cargo containers. 

In order to help improve the flow of commerce by eliminating many of the unnec- 
essary alarms that send cargo for secondary screening and to more accurately iden- 
tify radioactive or nuclear material, the Department of Homeland Security (DHS) 
began developing Advanced Spectroscopic Portals (ASPs) in 2004. The ASPs were 
intended to both detect and identify radioactive material. In April 2005, the Domes- 
tic Nuclear Detection Office was created by National Security Presidential Directive- 
43/Homeland Security Presidential Directive-14 to, among other things, research, 
develop, test and acquire radiation detection equipment to be used by CBP and 
other federal agencies. 

In July 2006, then-Secretary of Homeland Security Michael Chertoff and the 
former Director of DNDO, Vayl Oxford, announced contract awards to three compa- 
nies worth an estimated $1.2 billion to develop the ASPs, including the Raytheon 
Company and the Thermo Electron Company (now called Thermo Fisher Scientific 
Inc.) both headquartered in Waltham, Massachusetts and Canberra Industries from 
Connecticut. Canberra is no longer a contractor on the DNDO program. 

ASP Requirements/Criteria 

One of the key reasons for replacing the existing radiation monitors with newly 
developed ASPs in the first place, as articulated by Secretary of Homeland Security, 
Michael Chertoff in July 2006 was to “have fewer false positives.” In September 
2007, Vayl Oxford, then the director of DNDO reiterated that point in testimony to 
Congress where he emphasized that the ASPs would reduce the number of false 
alarms from the nearly 600 experienced each day by the PVTs at the port of Long 
Beach in California, for instance, to 20-to-25 per day with the new ASP monitors. 
That was the hope, but it has not been the reality during testing of the ASPs and 
other serious security questions about the performance reliability of the ASPs have 
emerged in the most recent round of tests. 

As the House Committee on Appropriations has said in the past, procurement of 
the Advanced Spectroscopic Portal monitors should not proceed until they are 
deemed to add a “significant increase in operational effectiveness” over the current 
PVT system already in place. In July 2008, CBP, DNDO and the DHS management 
directorate jointly issued criteria for determining this increase in effectiveness in 
both “primary” and “secondary” screening. In primary screening the criteria re- 
quires ASPs to detect potential threats as well as or better than PVTs, show im- 
proved detection of Highly Enriched Uranium and reduce innocent alarms. In sec- 
ondary screening the criteria requires ASPs to reduce the probability of 
misidentifying special nuclear material (HEU or plutonium) and reduce the average 
time to conduct secondary screenings. The Secretary of Homeland Security must cer- 
tify to Congress that the ASPs have met these criteria before funding for full-scale 
procurement of the ASPs goes forward. The criteria to measure this improvement, 
however, are weak and rather vague. 

Testing Regime 

Significant hurdles remain before ASPs can be certified and fully deployed. Both 
contractors have passed “integration testing.” They must now successfully make it 
through Field Validation Tests where they operate at ports of entry in tandem with 
PVT units. So far, only one of the two ASP vendors has made it to this stage. The 
one vendor that has made it to this stage will need to make its third attempt to 
successfully pass the Field Validation Tests before it can move forward. If and when 
they successfully pass this stage of testing they will then go to “Solo Operations,” 
where they will be tested at a port-of-entry operating independently of the PVTs. 
If they pass those two critical tests, then the DHS Directorate of Science & Tech- 
nology which has been mandated the Operational Testing Authority (OTA) of the 
ASPs will put them through a separate series of tests to ensure they meet the speci- 
fied requirements, do not suffer from technical glitches and operate efficiently. Once 
that testing is completed and the S&T Directorate signs off on the performance and 
reliability of the ASPs then the DHS Secretary must make a determination about 
whether the costs of the ASPs and the capabilities they provide justifies a decision 


VerDate Oct 09 2002 


09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00186 Fmt 6601 Sfmt 6621 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



181 


to invest in their full scale deployment. Along the way DNDO is supposed to provide 
a final cost-benefit-analysis of the ASP program to help inform the Secretary’s deci- 
sion. This document has been promised many times but not yet completed. 

Masking & Shielding 

If terrorists were to try to smuggle nuclear or radiological materials into the U.S. 
via containerized cargo they would likely try to shield and/or mask those materials 
in an attempt to make it more difficult to detect, identify and locate the material 
of concern. Shielding requires that lead or other types of metal enclose the 
radioisotopes to hide its radioactive signature. Potential terrorists may also attempt 
to “mask” threatening radioactive material by placing it together with or alongside 
other non-threatening material that has a natural radioactive signature, such as ce- 
ramic material, kitty liter or even bananas. Most nuclear security experts believe 
smuggled radioactive or nuclear material would be both shielded and masked in 
order to conceal it from being located and properly identified. These efforts would 
make it harder to detect. 

Many of DNDO’s previous tests of the ASPs have been criticized for being less 
than realistic. In one series of tests the ASP portals did prove more effective than 
the PVTs in detecting HEU materials concealed by “light shielding.” However, dif- 
ferences between the ASPs and PVTs became less notable when shielding was 
slightly increased or decreased. In other tests there was virtually no difference in 
the performance of the two machines with regard to detecting other kinds of radio- 
active isotopes, such as those used for medical or industrial purposes, according to 
the GAO, except in one case where the ASPs performed worse than the PVTs. In 
the most recent round of tests in July DNDO says the ASPs detected one radioactive 
source that the PVTs missed. 

In previous attempts to detect HEU during tests, the ASPs performed better only 
in one narrowly defined scenario, which many experts see as an unrealistic por- 
trayal of a true attempted nuclear smuggling incident. None of the tests run by 
DNDO, for instance, included scenarios that utilized both “shielding” and “masking” 
as a means of attempting to smuggle radioactive or nuclear material. In addition, 
only one of the vendors has made it to field validation testing. But as the contractor 
has attempted to fix problems that occurred during previous tests new, more serious 
technical issues have emerged. 

Field Validation Tests 

The Raytheon ASPs went through their first round of field tests last February, 
but technical issues hampered their performance. They had a large number of false 
alarms on several radioactive isotopes. Overall, in fact, the ASPs sent more cargo 
for secondary inspection than the currently operating PVTs did. Adjustments were 
made to prepare them for another round of field tests. Since the Subcommittee’s last 
hearing on the ASP program in June, the ASPs have gone through a second Field 
Validation Test at four U.S. ports of entry in L.A. Long Beach, California; the New 
York Container Terminal in Newark, New Jersey; Port Huron, Michigan; and La- 
redo, Texas. 

On average, the PVTs refer one out of every 40 cargo containers to secondary in- 
spection placing a large a burden on the staffing resources of CBP. The ASPs are 
required to send only one out of every 1,000 inspections to secondary inspection in 
order to help lessen that logistical burden. This is one of the key requirements that 
must be met in order for the Secretary of Homeland Security to permit full scale 
production of the ASPs to proceed. During the Field Validation Testing last Feb- 
ruary, however, the ASPs sent more than five times that number of cargo convey- 
ances to secondary inspection based on false alarms. During the most recent Field 
Validation Tests in July the ASPs reportedly reduced the number of false alarms 
compared to the PVTs by 69 percent bringing them much closer to the 80 percent 
reduction in false alarms that they are required to meet. But new, more serious 
problems also emerged during the field validation tests in July. 

During this second round of field tests the ASPs again failed to perform as ex- 
pected. This time they falsely identified several cargo conveyances as having special 
nuclear material, when they actually had none. This is a critical issue, since the ac- 
tual smuggling of special nuclear material presents a serious threat, if it is detected 
at a port-of-entry Customs and Border Protection officers have extensive response 
requirements they must implement. DNDO and the contractor are still unclear why 
the ASPs falsely identified special nuclear material during these tests. Their in- 
tended fix to this problem has been to decrease the sensitivity of the ASP monitors 
to specific radioactive isotopes. The hope is that this will correct the problem, reduce 
the number of false alarms and still ensure that the ASPs are able to detect these 
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isotopes. It is a delicate and difficult balance. It also decreases the ostensible advan- 
tage of having the ASPs replace the PVTs in the first place. 

Most unsettling, in one instance during the July tests one ASP monitor stopped 
working altogether yet the system operator remained unaware of this malfunction. 
Two dozen cargo trucks were permitted to go through the ASP in order to be 
screened for potential radioactive and nuclear material while it was not operating. 
DNDO considered this a “Mission Critical Failure.” Fortunately, during these tests 
all trucks that went through the ASP also went through a PVT monitor. If this had 
occurred during “solo” testing of the ASPs or during actual deployment of the ASPs, 
cargo carrying radiological or nuclear threat material could have sailed past port se- 
curity and into the United States unchecked. The cause of this problem has report- 
edly been rectified by the contractor. 

Energy Windowing 

Many experts believe significant improvements can be made to the existing fleet 
of PVT radiation monitors without investing billions of dollars into new ASPs. En- 
ergy windowing is a mathematical algorithm that can help improve the sensitivity 
of PVT radiation monitors, enhancing their ability to detect radioactive sources re- 
sulting in improved operations and capabilities. The technology is currently used in 
some radiation monitors. Both GAO and CBP believe that DNDO should much more 
aggressively invest in this research to improve the performance of the currently op- 
erating radiation detection monitors. Although energy windowing may only lead to 
modest enhancements in the performance of PVTs, that improvement could be sig- 
nificant in terms of improving their performance to be more on par with what ASPs 
are supposed to be capable of and at a far less financial cost. Reducing the sensi- 
tivity of the ASPs to certain types of special nuclear material, which was done to 
resolve the problems that emerged during the July tests, should not prevent them 
from alarming for isotopes that were not there in the first place. The only result 
would be to reduce the odds that the ASPs will identify those isotopes when they 
are actually present. 

A Dwindling Supply of Helium-3 (He-3) 

The future deployment of both PVT and ASP monitors is dependent on the supply 
of Helium-3 (He-3), a non-radioactive gas that is a byproduct of tritium decay. Trit- 
ium is a critical component in nuclear weapons used to boost the yield of nuclear 
warheads. Helium-3 gas is used in neutron detector tubes, a component of both PVT 
and ASP radiation portal monitors used to help identify plutonium. He-3 is also 
used in medical imaging, such as MRI machines, the oil and gas industry and for 
high energy research. During the cold war the U.S. had a steady supply of He-3 as 
a result of its nuclear weapons production operations. With the end of the cold war 
the production of nuclear weapons ceased and this supply diminished. At the same 
time, since 9/11 the demand for radiation monitors skyrocketed and demand for He- 
3 soon out-paced the supply. 

There are no readily available alternatives to He-3. In addition, no other tech- 
nology matches the stability, sensitivity, and ability to detect neutron radiation that 
He-3 neutron tubes currently offers. DNDO has estimated that the anticipated sup- 
ply-to-demand ratio of Helium-3 in coming years is expected to be l-to-10. Costs for 
the rare isotope have already begun to rise. By one estimate, a few years ago the 
cost of He-3 was around $100 per liter. Today, He-3 is estimated to cost as much 
as $2,000 per liter. According to a recent Department of Energy report, new ASP 
radiation monitors will use nearly three times more He-3 as current PVT monitors 
do, about 132 liters compared to 44 liters. These facts should be carefully considered 
by the Secretary of DHS when making cost-benefit decisions about whether or not 
to proceed with producing the ASPs. 

Cost Benefit Analysis 

Even if the technical abilities of the ASPs are proven, their relative technical ca- 
pabilities and increased costs must be carefully weighed in comparison to the exist- 
ing radiation monitoring system in place today. Replacing a proven, less-costly sys- 
tem that has the confidence of its operators, must be given careful consideration. 
The DNDO has not yet provided an updated cost-benefit-analysis that would vali- 
date a decision to procure the multi-billion dollar ASP equipment. 

Virtually any high-technology research and development program experiences 
bumps in the road, technical troubles and occasional set-backs. However, well man- 
aged programs have clear technical requirements and strategic goals. They ensure 
that the new technology being developed is thoroughly tested and adequately inte- 
grated into the operational plans and procedures of those who must operate them 
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in the field. When these vital components are short changed, when the test plan is 
insufficient and the program’s research, development and testing methods are 
marred by scanty scientific rigor then the technical tools being developed are bound 
to suffer as a result. Cutting critical corners in the development process serves no 
one’s interests. Yet, at the start of the ASP program many of the DNDO leaders 
seemed more interested in fielding this technology then in effectively validating its 
performance and effectiveness. At the July 2006 press conference unveiling the con- 
tractors on the ASP program, Vayl Oxford then the Director of DNDO said: “the pri- 
ority for the first year . . . is to get units out immediately.” Three years later, none 
of the ASPs have yet cleared field validation tests. 

Witnesses: 

Mr. Gene Aloise, Director, Natural Resources and Environment, Government Ac- 
countability Office (GAO) 

Dr. Timothy M. Persons, Chief Scientist, Government Accountability Office (GAO) 

Mr. Todd Owen, Executive Director for Cargo and Conveyance Security, U.S. Cus- 
toms and Border Protection (CBP), Department of Homeland Security (DHS) 

Dr. William Hagan, Acting Deputy Director, Domestic Nuclear Detection Office 
(DNDO), Department of Homeland Security (DHS) 
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Chairman Miller. Welcome to today’s hearing entitled The 
Science of Security, Part II: Technical Problems Continue to Hinder 
Advanced Radiation Monitors. 

Soon after the September 11 attacks, Customs and Border Pro- 
tection, CBP, began operating radiation portal monitors to screen 
cargo entering the United States for radiological and nuclear mate- 
rial. They have purchased approximately 1,500 polyvinyl toluene 
monitors and deployed them at ports and border crossings through- 
out the United States. Mercifully, polyvinyl toluene monitors are 
generally referred to as PVTs. 

PVTs indicate the presence of a radiation source, but they cannot 
identify the nature of the source. As a result, any cargo container 
that provokes a warning from a PVT is then sent to a secondary 
inspection where customs officers use other technology and infor- 
mation to determine what sort of material is in the container. 
There are plenty of innocent sources of radiation: kitty litter, med- 
ical isotopes, ceramics, bananas, and many of these secondary in- 
spections can be handled quickly. However, some secondary inspec- 
tions require that the container be opened, and even emptied, in 
the search for a source. That is time consuming, but the PVT 
seems to be working well to meet customs’ dual mission to keep us 
safe while maintaining a steady flow of commerce. 

The Department of Homeland Security (DHS) has been devel- 
oping a new radiation portal monitor, championed by the Domestic 
Nuclear Detection Office, DNDO, that advocates believe should re- 
place the PVTs. The new monitor, the Advanced Spectroscopic Por- 
tal monitor, or ASP, would detect the presence of radiation in 
cargo, but it would also identify the type of radiation. That would 
allow more harmless cargo to pass unimpeded through the port and 
require far fewer secondary referrals. If it worked as advertised, 
the ASPs would be more likely to identify highly enriched uranium 
or other materials of concern and enhance the flow of commerce 
while freeing up customs officers to tend to other duties besides 
secondary inspections. 

Despite a $230 million investment of taxpayer dollars thus far 
for development, the ASPs haven’t performed as expected, and the 
results from recent tests are still worrisome. Last June we learned 
of problems in the first field test in February 2009. At our June 
hearing we heard that those issues had been fixed and that the 
July field test would allow the department to move forward to- 
wards a cost-benefit analysis and certification decision for the Sec- 
retary. So far, we have seen neither. 

The July test highlighted yet another problem with the ASPs. 
The devices detected nuclear materials when none were present. 

The ASPs had numerous false positive hits for special nuclear 
material in July, each one of which would have resulted in the im- 
plementation of mandated security responses by Customs, poten- 
tially shutting down port operations. Fortunately, Customs was 
also running the PVTs which saw no radiation presence, and there 
was none, and cleared up the issue in secondary fairly quickly. 

DNDO has told our staff that they intend to fix this new problem 
by changing the sensitivity of the ASPs to detect uranium, but it 
isn’t clear why that should be reassuring, changing the sensitivity 
setting. If we lower the sensitivity setting to the very materials the 
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ASP is supposed to be better at monitoring or detecting, would the 
ASP still be better than PVTs at detecting those materials? And 
would we have to go back to the Nevada Test Site to prove that 
you can still detect levels of special nuclear material more accu- 
rately than the PVTs can? 

Second, if the machines detect special nuclear material when 
there is none there, how does changing the sensitivity make any 
difference at all? Detecting ghost isotopes is a problem with the op- 
erating system, not with the sensitivity level, it would appear any- 
way. 

Since the taxpayers have spent $200,000 for each of the 1,500 
PVTs that are deployed now, the case for ASPs, which will run 
about $800,000 per unit, a total cost of $2 to $3 billion, needs to 
be clear both in terms of better detection performance and better 
support for Customs operations. Add to that greater acquisition 
cost, an annual operating expense of ASPs that is at least five 
times more expensive per unit than PVTs, and the need for a con- 
vincing case is even greater. As it stands, it is hard to see why 
ASPs should be more than a secondary inspection tool. 

In fact, that is the role they play in the Department of Energy’s 
Megaport program. The DOE already runs a program that inspects 
cargo leaving 27 major foreign ports for destinations anywhere in 
the world. DOE, which developed portal radiation detection tech- 
nology, uses PVTs for primary inspection and then uses the ASPs 
for secondary inspection to help identify the type of isotope to 
which the PVT responded in the first place. The Department of En- 
ergy’s approach to identifying radiation should be instructive to 
DHS. 

I want to thank all of our witnesses for appearing today. I par- 
ticularly want to thank GAO for continuing their work on this mat- 
ter and for their continued assistance to this committee and Con- 
gress, and the Appropriations Committee as well. It is very helpful 
for the Appropriations Committee to be asking the right questions, 
and you have helped them and helped us ask the right questions. 
I suspect that this will not be the last time that we gather on this 
subject — we do seem to come back to the same subjects again and 
again — nor the last time we hear from witnesses that we still face 
a long list of tests and validations before we can think about re- 
placing the PVTs with the ASPs. 

I would now recognize the Ranking Member, Dr. Broun, for his 
opening comment. 

[The prepared statement of Chairman Miller follows:] 

Prepared Statement of Chairman Brad Miller 

Soon after the September 11 attacks, Customs and Border Protection (CBP) began 
operating radiation portal monitors to screen cargo entering the United States for 
radiological or nuclear material. They have purchased approximately 1,500 polyvinyl 
toluene (PVT) monitors and deployed them at ports and border crossings throughout 
the United States. 

PVTs indicate the presence of a radiation source, but they cannot identify the na- 
ture of the source. As a result, any cargo container that provokes a warning from 
a PVT is then sent to a “secondary” inspection where Customs officers use other 
technology and information to determine what sort of material is in the container. 
There are plenty of innocent sources of radiation — from kitty litter to medical iso- 
topes — and many of these secondary inspections can be handled quickly. However, 
some secondary inspections require that the container be opened, and even emptied, 
in the search for a source. While this is time consuming, the PVT seems to be work- 
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ing well to meet Customs’ dual mission to keep us safe while maintaining a steady 
flow of commerce. 

The Department of Homeland Security has been developing a new radiation por- 
tal monitor, championed by the Domestic Nuclear Detection Office (DNDO), that ad- 
vocates believe should replace the PVTs. This new monitor — the Advanced 
Spectroscopic Portal monitor or ASP — would detect the presence of radiation in 
cargo, but it would also identify the type of radiation. This would allow more harm- 
less cargo to pass unimpeded through the port and require far-fewer secondary re- 
ferrals. If it worked as advertised, the ASPs would be more likely to identify highly 
enriched uranium and other materials of concern and enhance the flow of commerce 
while freeing up Customs officers to tend to other duties besides secondary inspec- 
tions. 

Despite a $230 million investment of taxpayer dollars for development, the ASPs 
haven’t performed as expected, and the results from recent field tests are worri- 
some. Last June we learned of problems in the first field test of February 2009. At 
our June hearing we heard that those issues had been fixed and that the July field 
test would allow the Department to move towards a cost-benefit analysis and certifi- 
cation decision for the Secretary. To date, we have seen neither. 

The July field test highlighted yet another problem with the ASPs: the devices de- 
tected nuclear materials when none were present. 

The ASPs had numerous false positive hits for special nuclear material in July — 
each one of which would have resulted in the implementation of mandated security 
responses by Customs, potentially shutting down port operations. Fortunately, Cus- 
toms was also running the PVTs (which properly saw no radiation present) and 
cleared up the issue in secondary. 

DNDO has told our staff that they intend to fix this new problem by changing 
the sensitivity of the ASPs to detecting uranium, but it isn’t clear that should be 
reassuring. If you lower the sensitivity to the very materials the ASP was supposed 
to be better at detecting, why would the ASP will still be better than PVTs at de- 
tecting those materials? Would you have to go back to the Nevada Test Site to prove 
that you can still detect levels of special nuclear material more accurately than the 
PVTs can? 

Second, if the machines detect special nuclear material where it doesn’t exist, why 
should changing the sensitivity make any difference at all? Detecting ghost isotopes 
is a problem with the operating system itself, not with the sensitivity level for a 
particular isotope. 

Since the taxpayers have spent $200,000 for each of the 1,500 PVTs already de- 
ployed, the case for ASPs, which will run approximately $800,000 per unit — for a 
total cost of $2-3 billion — needs to be clear both in terms of better detection per- 
formance and better support for Customs operations. Add to this greater acquisition 
cost, an annual operating expense of ASPs that is at least five times more expensive 
per unit than PVTs, and the need for a convincing case is even greater. As it stands, 
it is hard to see why ASPs should be more than a secondary inspection tool. 

In fact, that is the role they play in the Department of Energy’s Megaport pro- 
gram. DOE already runs a program that inspects cargo leaving 27 major foreign 
ports for destinations anywhere in the world. DOE, which developed portal radiation 
detection technology, uses PVTs for primary inspection and reserves ASPs for sec- 
ondary inspections to help identify the type of isotope to which the PVT responded. 
The Department of Energy’s approach to identifying radiation should be instructive 
to DHS. 

I want to thank our witnesses for attending today. I particularly want to thank 
GAO for their continuing work on this matter and for their continuing assistance 
to this committee and Congress. I suspect that this will not be the last time we 
gather on this subject, nor the last time we hear from witnesses that we still face 
a long list of tests and validations before we can even speak sensibly about replacing 
PVTs with ASPs. 

Mr. Broun. Thank you, Mr. Chairman. I want to welcome the 
witnesses here today and thank you all for participating in our fol- 
low-up hearing on the Department of Homeland Security’s Ad- 
vanced Spectroscopic Portal program. It is hard for a Southerner 
to say quickly. 

This afternoon we will be brought up to date on the Depart- 
ment’s ongoing development of the next generation radiation portal 
monitors and get an update from the GAO on their continuing 
work. As I said in our earlier hearing this past summer, this pro- 
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gram is certainly not out of the woods. The latest field validation 
tests reveal additional problems that will have to be overcome be- 
fore moving forward. I hope DNDO will be able to give us some in- 
sight today on what we can expect from this program in terms of 
the future paths forward. With considerable taxpayer money on the 
line, questionable improvements over current capabilities and out- 
standing cost benefit analysis and a confusing acquisitions history 
that unfortunately has morphed the R&D with procurement, this 
program is rapidly approaching a point where the Federal Govern- 
ment has to decide whether it wants to fish or cut bait. I am con- 
cerned with the fact that considerable public funding has been ex- 
pended on developing a technology that the private sector was de- 
veloping in parallel on its own dime. DHS as a whole, and DNDO, 
CBP, DHS, S&T individually, should be focusing on long-term, 
high-risk, high-reward technology, not providing seed money for 
commercial, off-the-shelf equipment. That being said, I realize that 
DHS’s mission is vastly different from DOE’s, the Department of 
Defense’s and that they have additional requirements that demand 
a more robust system. GAO and the Academy made several rec- 
ommendations over the past few years. I trust that DNDO and 
CBP will be able to update this committee on how they are re- 
sponding to those recommendations and where they plan to go from 
here. The Nation expects a lot from the Department, and I hope 
that we aren’t developing tunnel vision by focusing too much on 
one method of conveyance and not seeing the forest for the trees. 
The Department has an enormous task of securing our borders and 
not just at points of entry but all along our borders. Spending bil- 
lions of dollars to secure the front door of our house doesn’t seem 
very rational if we are just going to leave the back door open and 
all the windows and have a gaping hole in the walls, too. That is 
not to say that we should do nothing at all, but rather, everything 
we do should be put in context of a well-thought-out global nuclear 
detection architecture. 

As I said earlier this summer, many of these issues we are deal- 
ing with today could have been prevented by engaging the end- 
users earlier in the process. Clearly defining the requirements, de- 
veloping clear architectural priorities and simply following a clear 
acquisition process. This Committee is no stranger to programs 
that have set aside these best practices for working in expediency’s 
sake. 

I look forward to working with the Department and the Majority 
to make sure any decision that is made is in the best interest of 
our nation’s security, the taxpayer and our economy. 

With that, Mr. Chairman, I yield back my time and I thank you. 

[The prepared statement of Mr. Broun follows:] 

Prepared Statement of Representative Paul C. Broun 

Thank you, Mr. Chairman. I want to welcome the witnesses here today, and 
thank them for participating in our follow-up hearing on the Department of Home- 
land Security’s (DHS) Advanced Spectroscopic Portal (ASP) program. This afternoon 
we will be brought up to date on the Department’s ongoing development of next gen- 
eration Radiation Portal Monitors and get an update from the General Account- 
ability Office (GAO) on their continuing work. 

As I said at our earlier hearing this past summer, this program is certainly not 
out of the woods. The latest Field Validation Test revealed additional problems that 
will have to be overcome before moving forward. I hope DNDO will be able to give 
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us some insight today on what we can expect from this program in terms of future 
paths forward. With considerable, taxpayer money on the line, questionable im- 
provements over current capabilities, an outstanding cost-benefit analysis, and a 
confusing acquisitions history that unfortunately has morphed Research and Devel- 
opment (R&D) with procurement, this program is rapidly approaching a point where 
the Federal Government has to decide to “fish or cut bait.” 

I’m also concerned with the fact that considerable public funding has been ex- 
pended on developing a technology that the private sector was developing in parallel 
on its own dime. DHS as a whole (and DNDO, CBP, and DHS S&T individually) 
should be focusing on long-term high-risk high-reward technology, not providing 
seed money for Commercial Off-The-Shelf (COTS) equipment. That being said, I re- 
alize that DHS’ mission is vastly different from the Department of Energy’s (DOE) 
and the Department of Defense’s (DOD), and that they have additional require- 
ments that demand a more robust system. 

GAO and the Academy made several recommendations over the last few years. 
I trust that DNDO and CBP will be able to update this committee on how they are 
responding to those recommendations, and where they plan to go from here. The 
Nation expects a lot from the Department, and I hope that we. aren’t developing 
tunnel vision by focusing too much on one method of conveyance and not seeing the 
forest through the trees. The Department has an enormous task of securing our bor- 
ders, not just at points of entry, but all along our borders. Spending billions of dol- 
lars to secure the front door or our house, doesn’t seem very rational if we are just 
going to leave the back door open: That is not to say we should do nothing at all, 
but rather everything we do should be put in the context of a well thought out Glob- 
al Nuclear Detection Architecture. 

As I said earlier this summer, many of the issues we are dealing with today could 
have been prevented by engaging the end-users earlier in the process, clearly defin- 
ing requirements, developing clear architectural priorities, and simply following a 
clear acquisition process. This committee is no stranger to programs that have set 
aside these best practices for expediency’s sake. I look forward to working with the 
Department and the majority to make sure any decision made is in the best interest 
of our nation’s security, the taxpayer, and our economy. 

With that, Mr. Chairman, I yield back my time. 

Thank you. 

Chairman Miller. Thank you, Dr. Broun. I ask unanimous con- 
sent that all additional opening statements submitted by Members 
be included in the record. Without objection, it is so ordered. Also, 
there has been an e-mail exchange between our staff and the staff 
of DNDO, Kimberly Koeppel, and without objection, I move that 
the printed versions of the e-mailed questions and answers also be 
entered into the record. Without objection, so ordered. 

[The information follows:] 
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O'Rourke, Molly 


From: Houser, Jason [mailto:Jason.Houser@dhs.gov] 
Sent: Monday, November 16, 2009 12:20 PM 
To: Pasternak, Doug 
Subject: Fw: Possible 0's 

Fyi 

Jason P Houser 
jason houser@dhs gov 


From: Keep pel, Kimberly 

To: Houser, Jason 

Sent: Mon Nov 16 12:16:01 2009 

Subject: RE: Possible Q's 

Questions from House S&T Committee Stall' 


Q. It appears that radiation detectors can set oft' alarms at ports for a variety of reasons. It is our 
understanding that these are classed as ( 1 ) "innocent alarms.” which are alarms that result when NORM 
materials, such as granite, kitty litter, bananas, etc., go through the detectors; (2) “false alarms” or “false 
positives,” which are detection errors because there is no radioactive material in the containers being scanned; 
and (3) “real alarms.” which result from the presence of radioactive material that is of national security 
concern. 

.Are these definitions accurate? If not. please provide the correct definitions. 

Clarifications are: 

“False alarms” or “false positives'* include all occupancies that are misidentified as a “real alarm” 

A “real alarm” includes medical, industrial and high levels of NORM radiation (w hich can potentially hide 
SNM) as w ell as material of national security concern. 

Q. According to the information provided to staff last w eek, during tandem testing in January, the Raytheon 
.ASPs gave off “false alarms” - as defined above - for americium, thorium and cobalt. Is that accurate, or did 
the ASPs simply detect smaller amounts of these isotopes than w as necessary to meet specifications? 

The .ASP did give false alarms for those isotopes. 'Ilie threshold settings for those isotopes were set at a low 
level during the January Field Validation. Raising the thresholds reduced the false alarms as shown through 
replay and verified during July Field Validation. These changes kept the system w ithin specifications and did 
not impact SNM sensitivity. 

Q. DNDO staff also staled that there were some “false alarms” relating to special nuclear material in the 

January testing, but that they were “below concern.” What does it mean to be “below concern”? What did 
DNDO and' or Raytheon do about those? 

The ASP system must strike a balance between high sensitivity on materials of interest (e.g., SNM) and a false 
referral rate that is not an operational burden. To be “below concern.” the referral rate must not unduly burden 
the operators, yet maintained the required sensitivity. In July, against a different stream of commerce data set. a 
higher false alarm rate on SNM was observed. Consequently, that these threshold have been modified which 
will reduce this false alarm rate, while maintaining adequate detection capabilities. 

Q. List the specifications for innocent alarms and false alarms/false positives for the ASP. 
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Ltefiniton of INNOCENT Alarm, taken from Spec Y4.1Q; 

{ASP- 1370} GREEN = Innocent notification - Used to indicate that only NORM radionuclides have been 
identified. Generally does NOT select vehicle for additional inspection unless other negative factors are present. 

From Appendix 3 (Alarm Definitions) of the spec: 

Correct Dismissal — Correct Negative 

A correct dismissal is defined as properly releasing non-threat cargo into the stream-of-commerce without 
further inspection. For ASP. the GREEN (Innocent) and WHITE (No radioactivity) indications result in NO 
diversion to secondary for further investigation. 

...and also... 

e. Innocent Alarm State 

For the ASP, an innocent alarm slate (GREEN) is produced by conditions defined by the user to over-ride 
specified gamma gross-count alarm states. 

For example, an innocent alarm state exists when an over-threshold gross-count alarm identified as NORM 
without SNM occurs, and the user has set a condition to block the gross count alarm notification. The “innocent 
alarm slate” vehicle is not diverted to secondary screening and does not produce a false alarm because the 
potential gross-count false alarm was automatically cleared based on spectral information. It is desirable for 
potential “nuisance false alarm” vehicles to process as innocent alarm states. However, it is possible for SNM 
cargo to be passed as an innocent alarm state which is a false negative error. There is real risk that a simple 
portal or handheld spectrometer with only a small Nal crystal can readily respond to NORM cargo, but not be 
sensitive to threat materials surrounded by cargo. Therefore, rigorous testing to establish adequate sensitivity to 
SNM surrounded by cargo is required for any sensor system incorporating an innocent alarm state feature or 
being used to clear false alarms. 

In secondary screening, one desires sensitivity to NORM cargo and thus the ability to readily identify NORM 
causes of gamma gross count alarms generated by a PVT RP\1 in primary . The ASP identification a potential 
reason for a gamma gross count alarm is more than release on NORM excuse, because the ASP is capable of 
identifying threat radionuclides and these were not found during the ASP survey. 

Definition of False .Alarm, taken from Spec v4.1Q: 

(relative to gamma detector) 

{ASP-313} Gross counting alarm threshold shall be set with a statistical false alarm rate of < 0. 1%. 

(relative to neutron detector) 

{ASP-391} Gross counting alarm threshold shall be set with a false alarm rate of < 0. 1%. 

From Appendix 3 (Alarm Definitions) of the spec: 

False Alarm — False Positive Error 

In the context of vehicle surveys, a false alarm is defined as any threat alarm or diversion of a non-threatening 
vehicle from primary screening to secondary screening not due to a threat material. In statistical terms, this is a 
false positive error. A false alarm is due to one of the following: 

1) Due to either a statistical fluctuation defined as a statistical false alarm. 

2) Due to physical causes defined as a systematic false alarm. 

This false alarm discussion is in the framew ork of a radiation measurement producing a count value (e.g., gross 
count, energy-window count, or peak-region count). 

The systematic and statistical components of the measurement uncertainty are added in quadrature to produce 
an aggregate uncertainty. 
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a. Statistical False Alarm 

Statistical false alarms are a result of random statistical fluctuations in a measurement or count value. For 
Poisson statistics, the purely statistical standard error in the count value equals the square root of the count 
value. Whenever the statistical component dominates, the standard error scales with the square root of the 
product of the sensor area and measurement time. For spectral algorithms, the peak region 'is generally counted 
starved and the statistical component dominates. 

b. Systematic False Alarm 

Systematic false alarms are a result of changes or uncertainties in count rates due to physical causes. A nearby 
radiation source is a common cause of systematic false alarms. The systematic component is not readily 
estimated from a statistical model but can be empirically determined. The survey scenario includes some 
empirical distribution related to the number of vehicles carrying radiation sources and the size for those 
radiation sources. Physical causes may include NORM cargo, medical radiation sources, industrial radiation 
shipments, radon plumes, and variable shielding scenarios. Some radiation sources are associated with the 
surveyed vehicle and others with the surrounding environment. During vehicle surveys, most gross-count false 
alarms are systematic and are due to a physical cause from a radiation source within the vehicle. In this case, the 
systematic false alarm rate is not reduced by increasing the sensor size or the measurement lime. Spectral and 
temporal characteristics can identify and eliminate some undesirable physical cause scenarios. For gross 
counting, the systematic component dominates vehicle surveys, and the false alarm rate will not be reduced by 
increasing sensor size. 

c. Nuisance False Alarm 

For vehicle surveys, a nuisance false alarm is a systematic false alarm associated with some actual radiation 
source carried within the vehicle (e.g.. NORM cargo or medical radionuclide in a vehicle). If medical and 
industrial sources are included in the threat definition, they should not be considered a false alann cause. The 
declaration of an ASP GREEN (Innocent) indication or alarm for NORM-cargo does not result in diversion to 
secondary' or count as a false alann. because the ASP algorithm recognized the radiation source as acceptable 
for commerce. The declaration of a PVT or gross count GAMMA alarm due to NORM-cargo results in 
diversion to secondary and counts as a false alarm, because there was no indication that the radiation source is 
acceptable for commerce. To be defined as a nuisance or innocent false alann. the actual radiation source 
eausing the false alarm must be within the acceptable limits of norma! commerce. Acceptable radiation sources 
can be considered innocent or benign. Vehicles causing these systematic false alamis are eleared by secondary' 
inspection and returned to the slrcam-of-commcrcc with no enforcement action. 'Ilius. nuisance or innocent 
refers to the radiation content of the vehicle or physical cause of the alarm. 

... and later ... 

f. Non -Vehicle False' Alann 

A non-vehicle false alarm is defined as any false alann caused by sources external to the portal with a vehicle 
within the portal. Some surveys may produce a false alann without any physical cause being present within the 
sun eyed vehicle but the cause is a strong source located in the vicinity of the portal. Examples include a hot 
medical source in a nearby vehicle (i.e., cross-talk), a radon plume, and a nearby radiographic source. 


Q. Please provide side-by-side data for both the PVTs and ASPs for the January and July tests for the 
following categories: 

1. Total number of innocent alarms: how many would be released at primary' screening, and how many 
would have gone to secondary' under a solo ASP system. 

2. Number of false alarms and or false positives for special nuclear material. 

3. Number of real alamis that require referral to secondary screening. 
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• 

PVT January 

ASP January 

PVT July 


Innocent referred to 
secondary 

327 

0 

441 

0 

Innocent released 

unknown 

145 

unknown 

217 


•The PVT numbers include referrals from 2 lanes while the ASP numbers include only one lane. 


FV 3 


Radionuclide ID’d 

NYCT 

Feb 

2009 

False 

Alarms 

LA/LB 

Feb 

2009 

False 

Alarms 


Am-241 

13 

22 

35 

Cr-51 

0 

0 

0 

Co-60 

5 

3 

8 

TI-201 

3 

0 

3 

Cs-137 

2 

0 

2 

U-235 

9 

3 

12 

U-238 

4 

1 

5 

Pu-238 

1 

0 

1 

Pu-239 

0 

1 

1 


9 

8 

17 

TOTAL 

46 

38 

84 

TOTAL PRIMARY 
OCC 

10845 



% of unwanted 
alarms 

0 77% 



Real Alarms 


1 

1 

Gross Referral Rate 




PVT Gross Referral 
Rate 

TBD 




FV 3e 


Radionuclide ID’d 

NYCT 

Jul 

2009 

False 

Alarms 

LA/LB 

Jul 

2009 

False 

Alarms 

TOTAL 

Am-241 

0 

0 

0 

Cr-51 

0 

1 

1 

Co-60 

0 

1 

1 

TI-201 

0 

0 

0 

Cs-137 

0 

0 

0 

U-235 

15 

11 

26 

U-238 

16 

2 

18 

Pu-238 

0 

2 

2 

Pu-239 

1 

0 

1 


17 

18 

35 

TOTAL 

49 

35 

84 

TOTAL 

OCCUPANCIES 

12937 



% of unwanted 
alarms 

0 65% 



Real Alarms 

1 


1 

Gross Referral Rate 




PVT Gross referral 
Rate 

2 44% 




•The referral rate does not include over-speed occupancies. •• Based on preliminary analysis 
TBD^We are still getting the numbers for PVT gross referral rate for the Jan FV. 


Respectfully, 

'Kimberly (Stein) Koeppel 

Domestic Nuclear Detection Office 

U.S. Department of Homeland Security 

Office of the Director 

Desk: 202.254.7322 

Mobile: 202.368.4446 

HSDN: kimberly.koeppel@dhs.sgov.gov 


Chairman Miller. It is my pleasure to introduce our witnesses 
at this time. Mr. Gene Aloise is the Director of Natural Resources 
and Environment at the Government Accountability Office, GAO. 
He is an expert in international nuclear proliferation and safety 
issues and holds degrees in political science, economics and public 
administration. Mr. Aloise is a recipient of GAO’s Meritorious Serv- 
ice and Distinguished Services Awards and has served several Con- 
gressional committees and offices within GAO. This is not the first 
time he has appeared before us on this topic. 

With Mr. Aloise is Dr. Timothy Persons, GAO’s Chief Scientist. 
Before entering GAO, before joining GAO, Dr. Persons was the 
Technical Director of the Intelligence Advanced Research Projects 
Activity, I-ARPA, and previously served as a technical director at 
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the National Security Agency. Mr. Persons, if people ask you at a 
cocktail party what you did for a living, did you just say you 
worked for the government? He holds degrees in nuclear physics, 
computer science and biomedical engineering and has been in- 
volved in evaluations of many high-tech U.S. Government pro- 
grams and projects, including GAO’s work on the Advanced 
Spectroscopic Portals, ASP monitors. We look forward to hearing 
his testimony today as well. 

Mr. Todd C. Owen is the Acting Deputy Assistant Commissioner 
of the Office of Field Operations for the U.S. Customs and Border 
Protection, Department of Homeland Security. Also a mouthful. He 
also served as the Executive Director of the Cargo and Conveyance 
Security Office, Office of Field Operations, since May of 2006. Mr. 
Owen began his career with the Customs and Border Protection in 
1990 and has previously held the positions of Area Port Director 
in New Orleans and Director of the Customs Trade Partnership 
Against Terrorism, C-TPAT Program. 

And then finally, Dr. William Hagan is the Acting Deputy Direc- 
tor of the Domestic Nuclear Detection Office at the Department of 
Homeland Security. Dr. Hagan holds degrees in physics and nu- 
clear engineering as well as three patents. He spent 30 years with 
the Science Applications International Corporation before coming 
to DNDO and serving as the Assistant Director of the Trans- 
formational Research and Development Directorate. 

I think all of our witnesses would need to use small print for 
their business cards. 

As our witnesses should know, you will each have five minutes 
for your spoken testimony. Your written testimony will be included 
in the record for the hearing. When you all have completed your 
spoken testimony, we will begin with questions. Each Member will 
have five minutes to question the panel. It is the practice of this 
subcommittee to receive testimony under oath. This is an Inves- 
tigations and Oversight Subcommittee. Do any of you have an ob- 
jection to taking an oath? Let the record reflect that none of the 
witnesses had any objection. You also have the right to be rep- 
resented by counsel. Do any of you have counsel here? Let the 
record reflect that all of the witnesses said no. We ask you these 
questions to put you at ease. 

Would you all now please stand and raise your right hand? 

Mr. Broun. And we are not going to water board them. 

Chairman Miller. Well, that would be the next hearing. Please 
stand and raise your right hand. Do you swear to tell the truth and 
nothing but the truth? Let the record reflect that each of the wit- 
nesses did take the oath. 

We will begin with Mr. Gene Aloise. Mr. Aloise, please begin. 

STATEMENT OF MR. GENE ALOISE, DIRECTOR, NATURAL RE- 
SOURCES AND ENVIRONMENT, U.S. GOVERNMENT ACCOUNT- 
ABILITY OFFICE (GAO) 

Mr. Aloise. Mr. Chairman and Members of the Subcommittee, 
I am pleased to be here today to discuss DHS’s plans to develop 
and test advanced portal monitors, known as ASPs, for use at the 
Nation’s borders to prevent nuclear materials from being smuggled 
into the United States. My testimony today focuses on the results 
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of DNDO’s testing of the ASPs, including the July 2009 field test- 
ing. 

According to DHS, the current system of radiation detection 
equipment, the PVTs, is effective and does not impede the flow of 
commerce. However, DHS wants to improve the capabilities of the 
existing equipment with ASPs. One of the major drawbacks of the 
ASP, as you know, is a substantially higher cost compared to the 
existing equipment. We estimated that the life cycle cost of each 
standard cargo version of the ASP to be about $822,000 compared 
to about $308,000 for the PVT standard cargo portal and that the 
total program cost would be about $2 billion. 

Earlier this year I testified before this subcommittee on DNDO’s 
2008 round of ASP performance testing at the Nevada Test Site. 
That testing showed that the ASPs performed better than the PVTs 
in detecting certain nuclear materials and met DOE’s threat guid- 
ance. However, the ASP’s performance rapidly deteriorated once 
shielding was slightly increased. These test results showed what 
we reported in 2006, that any increase in detection of certain nu- 
clear materials would be marginal. 

While the ASP may have met DOE’s threat guidance for shielded 
nuclear material, the threat guidance is not a realistic approxima- 
tion of how a terrorist might shield nuclear material to successfully 
smuggle it undetected across our borders. 

The latest round of field testing conducted in July revealed two 
critical performance problems with ASPs that the Chairman men- 
tioned. First, the ASPs had an unacceptably high number of false 
positives for the detection of high-risk nuclear material. In other 
words, the ASP was seeing nuclear material that wasn’t there and 
alarming. CBP officials told us that any alarm for this type of nu- 
clear material is very disruptive to a port or border crossing and 
could effectively shut down operations until the source of the alarm 
is found. Furthermore, repeated false alarms for nuclear materials 
could have the undesired effect of causing CBP officers to doubt the 
reliability of the ASPs and be skeptical about the credibility of fu- 
ture alarms. 

The second critical failure of the ASPs noted in the July testing 
stemmed from a problem with the key component of the equipment 
which led an ASP to in essence shut down. Of great concern was 
the fact that the ASP did not alert the CBP official that it had shut 
down and was not scanning cargo. If this were not a controlled test, 
the CBP officer would have allowed the cargo to enter the United 
States thinking it had been scanned when it had not. 

DNDO’s proposed solutions to these critical failures raise ques- 
tions about whether the ASPs will provide any meaningful increase 
in the ability to detect certain nuclear materials. Specifically, to ad- 
dress the problem of false positives, DNDO is modifying the ASP 
to make it less sensitive to certain nuclear materials. While this 
may fix the problem of false alarms, it diminishes even further the 
ASP’s ability to detect the nuclear material we are most concerned 
about. 

To address the second failure, DNDO plans to, among other 
things, install an indicator light on the ASP that will alert CBP of- 
ficials that the ASP has a mission-critical failure. In our view, an 
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indicator light is not the solution. The ASP must be stable and se- 
cure enough to avoid these shut-downs. 

Furthermore, DNDO has not completed efforts to improve the 
PVTs to detect high-risk material through energy windowing. CBP 
has repeatedly urged the completion of this research, because an 
improved PVT could be the more cost-effective way to improve de- 
tecting certain nuclear materials and have a similar performance 
to a working ASP. 

In closing, the concerns raised by the results of the July 2009 
field testing provide even greater reason for DNDO to implement 
our recommendations from our May 2009 report. In particular, our 
recommendation that DNDO assess whether the ASPs meet the 
criteria for significant increase in operational effectiveness based 
on a valid comparison with the PVT’s full performance. This is es- 
pecially relevant given that ASPs seemingly will no longer be as ef- 
fective in detecting certain nuclear materials. 

Mr. Chairman, that concludes my remarks. Dr. Persons and I 
would be happy to respond to any questions you may have. 

[The prepared statement of Mr. Aloise follows:] 

Prepared Statement of Gene Aloise 

Mr. Chairman and Members of the Subcommittee: 

I am pleased to be here today to discuss GAO’s work on the Department of Home- 
land Security’s (DHS) testing of advanced spectroscopic portal (ASP) radiation detec- 
tion monitors. One mission of U.S. Customs and Border Protection (CBP), an agency 
within DHS, includes screening cargo and vehicles coming into this country for 
smuggled nuclear or radiological material that could be used in an improvised nu- 
clear device or radiological dispersal device (a “dirty bomb”). To screen cargo at 
ports of entry, CBP conducts primary inspections with radiation detection equip- 
ment called portal monitors — large stationary detectors through which cargo con- 
tainers and vehicles pass as they enter the United States. When radiation is de- 
tected, CBP conducts secondary inspections using a second portal monitor to confirm 
the original alarm and a hand-held radioactive isotope identification device to iden- 
tify the radiation’s source and determine whether it constitutes a threat. 

The polyvinyl toluene (PVT) portal monitors CBP currently uses for this screening 
can detect radiation but cannot identify the type of material causing an alarm. As 
a result, the monitors’ radiation alarms can be set off even by shipments of bananas, 
kitty litter, or granite tile because these materials contain small amounts of benign, 
naturally occurring radioactive material. To address the limitations of current-gen- 
eration portal monitors, DHS’s Domestic Nuclear Detection Office (DNDO) in 2005 
began to develop and test ASPs, which are designed to both detect radiation and 
identify the source. 1 DNDO hopes to use the new portal monitors to replace at least 
some PVTs currently used for primary screening, as well as PVTs and hand-held 
identification devices currently used for secondary screening. 

Since 2006, we have been reporting on issues associated with the cost and per- 
formance of the ASPs and the lack of rigor in testing this equipment. For example, 
we found that tests DNDO conducted in early 2007 used biased test methods that 
enhanced the apparent performance of ASPs and did not use critical CBP operating 
procedures that are fundamental to the performance of current hand-held radiation 
detectors. 2 In addition, in 2008 we estimated the life cycle cost of each standard 
cargo version of the ASP (including deployment costs) to be about $822,000, com- 
pared with about $308,000 for the PVT standard cargo portal, and the total program 
cost for DNDO’s latest plan for deploying radiation portal monitors — which relies on 


1 DNDO was established within DHS in 2005; its mission includes developing, testing, acquir- 
ing, and supporting the deployment of radiation detection equipment at U.S. ports of entry. CBP 
began deploying portal monitors in 2002, prior to DNDO’s creation, under the radiation portal 
monitor project. 

2 Combating Nuclear Smuggling: Additional Actions Needed to Ensure Adequate Testing of 
Next Generation Radiation Detection Equipment. GAO— 07-1247T, (Washington, D.C.: Sept. 18, 
2007). 
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a combination of ASPs and PVTs and does not deploy radiation portal monitors at 
all border crossings — to be about $2 billion. 3 

Concerned about the performance and cost of the ASP monitors, Congress re- 
quired the Secretary of Homeland Security to certify that the monitors will provide 
a “significant increase in operational effectiveness” before DNDO obligates funds for 
full-scale ASP procurement. 4 In response, CBP, DNDO, and the DHS management 
directorate jointly issued criteria for determining whether the new technology pro- 
vides a significant increase in operational effectiveness. The primary screening cri- 
teria require that the new portal monitors detect potential threats as well as or bet- 
ter than PVTs, show improved performance in detection of highly enriched uranium 
(HEU), and reduce by 80 percent the number of innocent alarms that are sent to 
secondary inspection. To meet the secondary screening criteria, the new portal mon- 
itors must reduce the probability of misidentifying special nuclear material (e.g., 
HEU and plutonium) and the average time to conduct secondary screenings. 

DNDO designed and coordinated a new series of tests, originally scheduled to run 
from April 2008 through September 2008, to determine whether the new portal 
monitors meet the certification criteria and are ready for deployment. Key phases 
of this round of testing include concurrent testing led by DNDO of the new and cur- 
rent equipment’s ability to detect and identify threats and of ASPs’ readiness to be 
integrated into operations for both primary and secondary screening at ports of 
entry; field validation testing led by CBP at four northern and southern border 
crossings and two seaports; and an independent evaluation, led by the DHS Science 
and Technology Directorate at one of the seaports, of the new portal monitors’ effec- 
tiveness and suitability. 

In May 2009, we reported on the results of the then-current round of ASP test- 
ing. 5 The findings from that report were based on completed tests and preliminary 
results available at the time. Testing on ASPs has continued since that report was 
issued. Today my testimony will (1) discuss the principal findings and recommenda- 
tions from our May report and (2) update those findings based on the results of 
DNDO’s July 2009 ASP field validation testing. The findings we are presenting 
today are based on our previous ASP reports and updated with information collected 
during interviews with DNDO and CBP officials. We also reviewed testing results 
in a report on the July 2009 tests from the ASP Field Validation Advisory Panel, 
a panel made up of officials from CBP, DNDO, and a national laboratory established 
to examine testing results and provide recommendations. On November 12, 2009, 
we briefed DHS, CBP, and DNDO officials on the findings of our updated work. 
During the briefing, CBP and DNDO officials provided oral comments and offered 
additional information and clarifications we included in this testimony as appro- 
priate. Both our prior work and our updated work were conducted in accordance 
with generally accepted government auditing standards. Those standards require 
that we plan and perform the audit to obtain sufficient, appropriate evidence to 
produce a reasonable basis for our findings and conclusions based on our audit ob- 
jectives. We believe that the evidence obtained provides a reasonable basis for our 
statement today. 

Improved Testing Rigor Discussed in Our May 2009 Report Demonstrates 
Limitations of ASPs 

Our May 2009 report on the then-current round of ASP testing found that DHS 
increased the rigor of ASP testing over that of previous tests, and that a particular 
area of improvement was in the performance testing at the Nevada Test Site, where 
DNDO compared the capability of ASP and current-generation equipment to detect 
and identify nuclear and radiological materials. For example, unlike in prior tests, 
the plan for the 2008 performance test stipulated that the contractors who devel- 
oped the equipment would not be involved in test execution. This improvement ad- 
dressed concerns we previously raised about the potential for bias and provided in- 
creased credibility to the results. Nevertheless, based on the following factors, in our 


3 Combating Nuclear Smuggling: DHS’s Program to Procure and Deploy Advanced Radiation 
Detection Portal Monitors Is Likely to Exceed the Department’s Previous Cost Estimates. GAO- 
08-1108R, (Washington, D.C.: Sept. 22, 2008). 

4 Consolidated Appropriations Act, 2008, Pub. L. No. 110-161, 121 Stat. 1844, 2069 (2007); 
Consolidated Security, Disaster Assistance, and Continuing Appropriations Act, 2009, Pub. L. 
No. 110-329, 121 Stat. 3574, 3679 (2008); Department of Homeland Security Appropriations Act, 
2010, Pub. L. No. 111-83, 123 Stat. 2142, 2167 (2009). 

5 Combating Nuclear Smuggling: DHS Improved Testing of Advanced Radiation Detection Por- 
tal Monitors, hut Preliminary Results Show Limits of the New Technology, GAO— 09-655 (Wash- 
ington, D.C.: May 21, 2009). 
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report we questioned whether the benefits of the new portal monitors justify the 
high cost: 

• The DHS criteria for a significant increase in operational effectiveness. Our 
chief concern with the criteria is that they require only a marginal improve- 
ment over current-generation portal monitors in the detection of certain 
weapons-usable nuclear materials during primary screening. DNDO considers 
detection of such materials to be a key limitation of current-generation portal 
monitors. The marginal improvement required of ASPs to meet the DHS cri- 
teria is problematic because the detection threshold for the current-generation 
portal monitors does not specify a level of radiation shielding that smugglers 
could realistically use. Officials from the Department of Energy (DOE), which 
designed the threat guidance DHS used to set the detection threshold, and 
national laboratory officials told us that the current threshold is based not on 
an analysis of the capabilities of potential smugglers to take effective shield- 
ing measures but rather on the limited sensitivity of PVTs to detect anything 
more than certain lightly shielded nuclear materials. DNDO officials acknowl- 
edge that both the new and current-generation portal monitors are capable 
of detecting certain nuclear materials only when unshielded or lightly shield- 
ed. The marginal improvement in detection of such materials required of 
ASPs is particularly notable given that DNDO has not completed efforts to 
fine-tune PVTs’ software using a technique called “energy windowing” that 
could improve the PVTs’ sensitivity to nuclear materials. DNDO officials ex- 
pect they can achieve small improvements in sensitivity through energy 
windowing, but DNDO has not yet completed efforts to fine-tune the PVTs’ 
software. In contrast to the marginal improvement required in detection of 
certain nuclear materials, the primary screening requirement to reduce the 
rate of innocent alarms by 80 percent could result in hundreds of fewer sec- 
ondary screenings per day, thereby reducing CBP’s workload. In addition, the 
secondary screening criteria, which require ASPs to reduce the probability of 
misidentifying special nuclear material by one-half, address the limitations of 
relatively small hand-held devices in consistently locating and identifying po- 
tential threats in large cargo containers. 

• Results of performance testing and field validation. The results of performance 
tests that DNDO presented to us were mixed, particularly in the ASPs’ capa- 
bility to detect certain shielded nuclear materials during primary screening. 
The results of performance testing at the Nevada Test Site showed that the 
new portal monitors detected certain nuclear materials better than PVTs 
when shielding approximated DOE threat guidance, which is based on light 
shielding. In contrast, differences in system performance were less notable 
when shielding was slightly increased or decreased: both the PVTs and ASPs 
were frequently able to detect certain nuclear materials when shielding was 
below threat guidance, and both systems had difficulty detecting such mate- 
rials when shielding was somewhat greater than threat guidance. With re- 
gard to secondary screening, ASPs performed better than hand-held devices 
in identification of threats when masked by naturally occurring radioactive 
material. However, the differences in the ability to identify certain shielded 
nuclear materials depended on the level of shielding, with increasing levels 
appearing to reduce any ASP advantages over the hand-held identification de- 
vices. Other phases of testing uncovered multiple problems in meeting re- 
quirements for successfully integrating the new technology into operations at 
ports of entry. Of the two ASP contractors participating in the current round 
of testing, one has fallen behind due to severe problems encountered during 
testing of ASPs’ readiness to be integrated into operations at ports of entry 
(“integration testing”); the problems may require that the vendor redo pre- 
vious test phases to be considered for certification. The other vendor’s system 
completed integration testing, but CBP suspended field validation testing in 
January 2009 after two weeks because of serious performance problems re- 
sulting in an overall increase in the number of referrals for secondary screen- 
ing compared with existing equipment. 

• DNDO’s plans for computer simulations. As of May 2009, DNDO did not plan 
to complete injection studies — computer simulations for testing the response 
of ASPs and PVTs to simulated threat objects concealed in cargo containers — 
prior to the Secretary of Homeland Security’s decision on certification even 
though delays to the ASP test schedule have allowed more time to conduct 
the studies. According to DNDO officials, injection studies address the inabil- 
ity of performance testing to replicate the wide variety of cargo coming into 
the United States and the inability to place special nuclear material and 
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other threat objects in cargo during field validation. DNDO had earlier indi- 
cated that injection studies could provide information comparing the perform- 
ance of the two systems as part of the certification process for both primary 
and secondary screening. However, DNDO subsequently decided that per- 
formance testing would provide sufficient information to support a decision on 
ASP certification. DNDO officials said they would instead use injection stud- 
ies to support effective deployment of the new portal monitors. 

• Lack of an updated cost-benefit analysis. DNDO had not updated its cost-ben- 
efit analysis to take into account the results of ASP testing. An updated anal- 
ysis that takes into account the testing results, including injection studies, 
might show that DNDO’s plan to replace existing equipment with ASPs is not 
justified, particularly given the marginal improvement in detection of certain 
nuclear materials required of ASPs and the potential to improve the current- 
generation portal monitors’ sensitivity to nuclear materials, most likely at a 
lower cost. DNDO officials said they were updating the ASP cost-benefit anal- 
ysis and planned to complete it prior to a decision on certification by the Sec- 
retary of Homeland Security. 

Our May report recommended that the Secretary of Homeland Security direct 
DNDO to (1) assess whether ASPs meet the criteria for a significant increase in 
operational effectiveness based on a valid comparison with PVTs’ full performance 
potential and (2) revise the schedule for ASP testing and certification to allow suffi- 
cient time for review and analysis of results from the final phases of testing and 
completion of all tests, including injection studies. We further recommended that, 
if ASPs are certified, the Secretary direct DNDO to develop an initial deployment 
plan that allows CBP to uncover and resolve any additional problems not identified 
through testing before proceeding to full-scale deployment. DHS agreed to a phased 
deployment that should allow time to uncover ASP problems but disagreed with 
GAO’s other recommendations, which we continue to believe remain valid. 

Results from July 2009 Testing Raise Continuing Issues 

The results of DNDO’s most recent round of field validation testing, which it un- 
dertook in July 2009, after our May report was released, raise new issues. In July 
2009, DNDO resumed the field testing of ASPs at four CBP ports of entry that it 
initiated in January 2009 but suspended because of serious performance problems. 
However, the July tests also revealed ASP performance problems, including two crit- 
ical performance deficiencies. First, the ASP monitors had an unacceptably high 
number of false positive alarms for the detection of certain high-risk nuclear mate- 
rials. According to CBP officials, these false alarms are very disruptive in a port en- 
vironment in that any alarm for this type of nuclear material would cause CBP to 
take enhanced security precautions because such materials (1) could be used in pro- 
ducing an improvised nuclear device and (2) are rarely part of legitimate or routine 
cargo. Furthermore, once receiving an alarm for this type of nuclear material, CBP 
officers are required to conduct a thorough secondary inspection to assure them- 
selves that no nuclear materials are present before permitting the cargo to enter 
the country. Repeated false alarms for nuclear materials are also causes for concern 
because such alarms could eventually have the effect of causing CBP officers to 
doubt the reliability of the ASP and be skeptical about the credibility of future 
alarms. 

Secondly, during the July testing the ASP experienced a “critical failure,” which 
stemmed from a problem with a key component of the ASP and caused the ASP to 
shut down. Importantly, during this critical failure, the ASP did not alert the CBP 
officer that it had shut down and was no longer scanning cargo. As a result, were 
this not in a controlled testing environment, the CBP officer would have permitted 
the cargo to enter the country thinking the cargo had been scanned, when it had 
not. According to CBP officials, resolving this issue is important in order to assure 
the stability and security of the ASP. 

In addition to these key performance problems, the ASP was not able to reduce 
referrals to secondary inspection by 80 percent as required by the DHS criteria for 
a significant increase in operational effectiveness. According to the report from the 
ASP Field Validation Advisory Panel, a panel made up of officials from CBP, DNDO, 
and a national laboratory, the ASP was able to reduce referrals to secondary inspec- 
tion by about 69 percent rather than the 80 percent as required by the DHS criteria. 

While the performance of the ASP during the July field validation testing raises 
issues about its potential readiness for deployment, DNDO’s proposed solutions to 
address these performance problems raise additional questions about whether this 
equipment will provide any meaningful increase in the ability to detect certain nu- 
clear materials. Specifically, to address the problem of false positive alarms indi- 


VerDate Oct 09 2002 


09:19 Mar 17, 2010 Jkt 000000 PO 00000 Frm 00204 Fmt 6601 Sfmt 6621 Z:\DOCS\50324.TXT SCIENCE1 PsN: COMSCI 



199 


eating the presence of certain nuclear materials, according to DNDO officials, DNDO 
has modified the ASP to make this equipment less sensitive to these nuclear mate- 
rials. While this may address the issue of false positive alarms, it will also diminish 
the ASP capability of detecting a key high-risk nuclear material. Since the ASP 
modification, DNDO conducted computer simulations using a vendor-provided sys- 
tem called a “replay tool” to examine the effect of the modification on the ASP’s per- 
formance. According to DNDO officials, the replay tool demonstrated that the modi- 
fied ASP will still be able to detect certain nuclear materials better than the PVT. 
However, at this point, DNDO does not plan to retest the ASP at the Nevada Test 
Site where it can examine the effects of these modifications using actual nuclear 
materials. As we reported earlier this year, the results of the testing at the Nevada 
Test Site demonstrated that the ASPs represented a marginal improvement in de- 
tecting certain nuclear materials. By reducing the sensitivity to these materials and 
not retesting the modified ASPs against actual nuclear materials, it is uncertain ex- 
actly what improvement in detecting certain nuclear materials these costly portal 
monitors are providing. 

While DNDO is reducing the sensitivity of ASPs to certain nuclear materials, it 
has yet to complete efforts to improve the PVT’s ability to detect these same mate- 
rials through energy windowing. For several years, CBP officials have repeatedly 
urged DNDO officials to complete this research. However, it was not apparent from 
our discussions with DNDO officials if this effort is making meaningful progress 
with the development of energy windowing or when it will be completed. Further- 
more, CBP officials stated that, depending on the outcome of this research, energy 
windowing could be the more cost effective way to improve detection of certain nu- 
clear materials. In our view, ASPs being modified to diminish their capabilities to 
detect certain nuclear materials raises questions about whether energy windowing 
might be able to achieve a similar level of performance against these same materials 
from the PVTs that are already in place. 

Beyond reducing the sensitivity of ASPs to certain nuclear materials, DNDO also 
plans to address the issue of critical failures by, among other things, installing an 
indicator light on the ASP that will alert CBP officers that the ASP has experienced 
a mission-critical failure and is no longer scanning cargo. While this should address 
the issue of CBP officers not knowing that the ASP has suffered a critical failure, 
CBP officials stressed to us the need for the ASP to be stable and secure enough 
to avoid these shutdowns. 

In closing, the issues raised by the results of the July 2009 field validation tests 
provide even greater reason for DNDO to address recommendations from our May 
2009 report. In particular, we reiterate the importance of our prior recommendation 
for DNDO to assess whether ASPs meet the criteria for a significant increase in 
operational effectiveness based on a valid comparison with PVTs’ full performance 
potential, given that the ASPs will no longer be as effective in detecting certain nu- 
clear materials. 

Mr. Chairman, this completes my prepared statement. I would be happy to re- 
spond to any questions that you or other Members of the Subcommittee may have 
at this time. 
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STATEMENT OF DR. TIMOTHY M. PERSONS, CHIEF SCIENTIST, 
U.S. GOVERNMENT ACCOUNTABILITY OFFICE (GAO) 

Chairman Miller. Thank you, Mr. Aloise. Dr. Persons. I think 
I said Parsons earlier. Persons. You are here to answer questions 
but the testimony of Mr. Aloise is your testimony as well? 

Dr. Persons. That is correct. 


Biography for Timothy M. Persons 

Dr. Timothy M. Persons was appointed the Chief Scientist of the United States 
Government Accountability Office (GAO — the investigative arm of the U.S. Con- 
gress) in July of 2008. In this role he is an expert advisor and chief consultant to 
the GAO, Congress, and other federal agencies and government programs on cut- 
ting-edge science and technology (S&T), key highly-specialized national and inter- 
national systems, engineering policies, best practices, and original research studies 
in the fields of engineering, computer, and the physical and biological sciences to 
ensure efficient, effective, and economical use of science and technology in govern- 
ment programs. The Chief Scientist also works with GAO’s Chief Technologist to 
lead the production of Technology Assessments for the U.S. Congress. 

Prior to joining GAO, Dr. Persons was the Technical Director of the Intelligence 
Advanced Research Projects Activity (IARPA) from November 2002 to July 2008. 
IARPA’s mission is to invest in high-risk/high-payoff research with potential to revo- 
lutionize the business of intelligence collection, processing, analysis, and dissemina- 
tion. From July 2001 to November of 2002, he served as the Technical Director for 
the National Security Agency’s (NSA) Human Interface Security Group which re- 
searches, designs, and tests next-generation biometric identification and authentica- 
tion systems. He has also served as a radiation physicist with the University of 
North Carolina at Chapel Hill. 

Dr. Persons is a 2007 Director of National Intelligence S&T Fellow whose re- 
search focuses on computational imaging systems. He was also selected as the 
James Madison University (JMU) Physics Alumnus of 2007. He received his B.Sc. 
(Physics) from JMU, a M.Sc. (Nuclear Physics) from Emory University, and a M.Sc. 
(Computer Science) and Ph.D. (Biomedical Engineering) degrees from Wake Forest 
University. He is a senior member of the Institute for Electrical and Electronic En- 
gineers, Association for Computing Machinery, and the Sigma Xi research honor so- 
ciety and has authored an array of journal, conference, and technical articles. He 
is also a Ruling Elder in the Presbyterian Church in America and lives happily in 
Maryland with his wife Gena and their two children. 

Chairman Miller. Okay. Mr. Owen. 

STATEMENT OF MR. TODD C. OWEN, ACTING DEPUTY ASSIST- 
ANT COMMISSIONER, OFFICE OF FIELD OPERATIONS, U.S. 

CUSTOMS AND BORDER PROTECTION, DEPARTMENT OF 

HOMELAND SECURITY 

Mr. Owen. Chairman Miller, Ranking Member Broun, distin- 
guished Members of the Subcommittee, I am honored to be here 
this afternoon to provide an update on the role that U.S. Customs 
and Border Protection plays in protecting our nation from the illicit 
introduction of radiological or nuclear materials in cargo con- 
tainers, and on the future role that Advanced Spectroscopic Portal 
technology would have on CBP operations. 

I would like to begin by expressing my gratitude to Congress for 
its continued support toward CBP initiatives. Among the numerous 
priorities that were recognized in the American Recovery and Rein- 
vestment Act of 2009, Congress provided CBP with $100 million of 
stimulus funding toward upgrading and adding non-intrusive in- 
spection equipment. This funding will allow CBP to upgrade and 
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expand its successful Nil program and more effectively inspect con- 
tainers and vehicles crossing our border, allowing them to enter 
our country and its commerce in a safe and prompt manner. 

Everyday over 57,000 maritime containers, truck trailers and rail 
cars cross our border. CBP uses a multi-layered approach to ensure 
the integrity of supply chains from points of stuffing through ar- 
rival at U.S. ports of entry. This multi-layer defense is built upon 
interrelated initiatives which include the 24-Hour Rule and the 
Trade Act of 2002, the Automated Targeting System, Non-Intrusive 
Inspection (Nil) equipment and radiation portal monitors, the Con- 
tainer Security Initiative and the Customs Trade Partnership 
Against Terrorism, C-TPAT program. These complementary layers 
enhance security and protect our nation. 

Prior to 9—11, not a single radiation portal monitor and only 64 
large-scale Nil systems were deployed to our nation’s borders. By 
October of 2002, CBP had deployed the first RPM to the Ambas- 
sador Bridge in Detroit, and today CBP has just under 1,400 RPMs 
and 232 large-scale Nil systems deployed nationwide, and that is 
an increase of almost 200 more RPMs and five additional large- 
scale Nil systems since I last testified before this subcommittee in 
June. 

Nil technology allows the officers to detect possible anomalies, 
anomalies which may indicate the presence of weapons of mass ef- 
fect or some other contraband. In Fiscal Year 2009, CBP conducted 
over 5.2 million examinations using Nil technology, allowing CBP 
to meet our twin goals of enhance security and trade facilitation. 

In addition to the significant strides made in the area of Nil 
equipment, the CBP also continues to deploy first-generation radi- 
ation portal monitors to our nation’s ports of entry. Since last testi- 
fying before this subcommittee, CBP and DNDO have now com- 
pleted deployments of RPMs along the northern border and now 
scan 100 percent of trucks and passenger vehicles arriving from 
Canada, 100 percent arriving from Mexico and 98 percent of arriv- 
ing sea containers for the presence of nuclear radiological mate- 
rials. Since the first RPM was deployed in 2002, CBP officers have 
scanned over 404 million conveyances for the presence of radiation 
and 2.2 million radiological alarms have been successfully adju- 
dicated with minimal or no impact on the flow of legitimate trade 
and travel. CBP continues to closely coordinate with key stake- 
holders to ensure the impacts of this activity causes minimal dis- 
ruption on port operations. 

The first generation RPM systems, although very sensitive, do 
have limitations. While they alert the CBP officers to the presence 
of radiation, a secondary exam is necessary to positively identify 
the specific isotope causing the alert. In the event that a CBP offi- 
cer is unable to positively resolve the alarm, scientific reach-back 
is available 24 hours a day, seven days a week. 

The ASP is expected to enhance our detection capability while 
significantly reducing the number or secondary examinations. This 
is due to its ability to distinguish between actual threats and nat- 
ural or medical radiation sources that are not security threats. 
CBP has worked closely with the DNDO in the development and 
operational testing of ASP, has provided DNDO with functional re- 
quirements and has been actively engaged in every step of systems 
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evaluation. CBP will continue to work with DNDO, and with the 
DHS Science and Technology Directorate, S&T, toward Secretarial 
Certification, and is also working with DHS to ensure that any fu- 
ture acquisition and deployment decisions are consistent with DHS 
priorities. The decision to purchase and deploy ASPs in the oper- 
ational arena will be based on CBPs mission needs and operational 
requirements, a comprehensive cost benefit analysis to include a 
full understanding of the maintenance and operational cost and 
analysis of alternatives and other considerations. 

Mr. Chairman and Members of the Subcommittee, today I have 
addressed CBP’s commitment to investing in new and emerging de- 
tection technology aimed at enhancing cargo security. We must 
continue to maintain our tactical edge by integrating new tech- 
nology into our ports of entry. 

Thank you again for the opportunity to testify. I would be happy 
to answer any questions. 

[The prepared statement of Mr. Owen follows:] 

Prepared Statement of Todd C. Owen 

Chairman Miller, Ranking Member Broun, esteemed Members of the Sub- 
committee, it is a privilege and an honor to appear before you today to discuss the 
work of U.S. Customs and Border Protection (CBP), particularly the detection of ra- 
dioactive and nuclear material in cargo containers and the potential future role that 
the Advanced Spectroscopic Portal (ASP) program will have on our operations. CBP 
is responsible for ensuring the security of cargo entering the United States at our 
borders and facilitating the flow of legitimate trade and travel. As part of this proc- 
ess, over 98 percent of all arriving maritime containerized cargo is currently 
scanned for radiation through radiation portal monitors. 

I want to begin by expressing my gratitude to Congress for its continued support 
for the mission and people of CBP. It is clear that Congress is committed to pro- 
viding CBP the resources we need in order to increase and maintain the security 
of our borders. We appreciate your efforts and assistance. 

CBP is the largest uniformed federal law enforcement agency in the country. We 
station over 21,000 CBP officers at access points around the Nation, including air, 
land, and sea ports. We have deployed over 20,000 Border Patrol agents between 
the ports of entry. These forces are supplemented with 1,266 Air and Marine agents, 
2,392 agricultural specialists, and other professionals. 

CBP’s responsibilities include stemming the illegal flow of drugs, contraband and 
people, protecting our agricultural and economic interests from harmful pests and 
diseases, protecting American businesses from theft of their intellectual property, 
enforcing textile agreements, tracking import safety violations, regulating and facili- 
tating international trade, collecting import duties, facilitating legitimate travel, 
and enforcing U.S. trade laws. CBP facilitates lawful immigration, welcoming visi- 
tors and new immigrants, while making certain those entering this country are in- 
deed admissible and taking appropriate action when an individual fears being per- 
secuted or tortured if returned to their home country. At the same time, our employ- 
ees maintain a vigilant watch for terrorist threats. In FY 2009, CBP processed more 
than 361 million pedestrians and passengers, 109 million conveyances, 25.8 million 
trade entry summaries, examined 5.2 million sea, rail, and truck containers, per- 
formed over 26.8 million agriculture inspections, apprehended over 732,000 illegal 
aliens between our ports of entry, encountered over 224,000 inadmissible aliens at 
the ports of entry, and seized more than 2.8 million pounds of illegal drugs. 

We must perform our important security and trade enforcement work without sti- 
fling the flow of legitimate trade and travel that is so important to our nation’s 
economy. These are our twin goals: border security and facilitation of legitimate 
trade and travel. 

CBP OVERVIEW 

I am pleased to appear before the Subcommittee today to highlight the continued 
progress on radiation detection technology. I would also like to take this opportunity 
to bring attention to CBP’s holistic cargo security programs that are applied to all 
environments. CBP has made tremendous progress toward securing the supply 
chains that bring goods into the United States from around the world, and pre- 
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venting their potential use by terrorist groups, by: using cutting-edge technology to 
increase the ability of front-line CBP Officers to successfully detect and interdict il- 
licit importations of nuclear and radiological materials; moving resources where 
they are most needed; integrating all CBP offices; sharing information, including ac- 
tionable intelligence, across all aspects of CBP; and utilizing a multi-layered ap- 
proach to ensure the integrity of the supply chain from the point of stuffing, through 
arrival at a U.S. port of entry. This multi-layered approach includes the following 
comprehensive cargo security programs that are applied to all modes of transpor- 
tation: 

• Advance Information 

° 24-Hour Rule 

° Automated Targeting Systems 

o Importer Security Filing (commonly known as “10+2”) 

• The Customs Trade Partnership Against Terrorism (C-TPAT) and Free and 
Secure Trade (FAST) 

• Container Security Initiative (CSI) 

• Secure Freight Initiative 

• Use of non-intrusive inspection technology and mandatory exams for all high 
risk shipments. 

I will discuss each one of these layers in greater detail with particular focus on 
our radiation and nuclear detection capabilities. 

ADVANCE INFORMATION 

CBP requires advanced electronic cargo information, as mandated in the Trade 
Act of 2002, for all inbound shipments in all modes of transportation. This advanced 
cargo information is evaluated using the Automated Targeting System (ATS) before 
the cargo arrives in the United States. 

ATS provides decision support functionality for CBP officers working in Advanced 
Targeting Units at our ports of entry and Container Security Initiative ports abroad. 
The system provides uniform review of cargo shipments for identification of the 
highest threat shipments, and presents data in a comprehensive, flexible format to 
address specific intelligence threats and trends. Through rules, the ATS alerts the 
user to data that meets or exceeds certain pre-defined criteria. National targeting 
rule sets have been implemented in ATS to provide threshold targeting for national 
security risks for all modes of transportation: sea, truck, rail, and air. The DHS 
Science and Technology Directorate is exploring additional methodologies for con- 
ducting risk assessment. 

The Importer Security Filing interim final rule, also more commonly known as 
“10+2,” went into effect earlier this year and has already yielded promising results. 
This program will increasingly provide CBP timely information about cargo ship- 
ments that will enhance our ability to detect and interdict high risk shipments. 
Comments on aspects of this rule were accepted until June 1, 2009, and implemen- 
tation using informed compliance will continue until January of next year. Ship- 
ments determined by CBP to be high-risk are examined either overseas as part of 
our Container Security Initiative, or upon arrival at a U.S. port. 

CUSTOMS TRADE PARTNERSHIP AGAINST TERRORISM (C-TPAT) 

CBP works with the trade community through the Customs Trade Partnership 
Against Terrorism (C-TPAT) to better secure goods moving through the inter- 
national supply chain. C-TPAT has enabled CBP to leverage supply chain security 
throughout international locations where CBP has no regulatory reach. Under the 
C-TPAT program, a prospective member submits basic company information and a 
security profile via an Internet-based portal system. CBP conducts records checks 
on the company in its law enforcement and trade databases and evaluates the secu- 
rity profile, ensuring the company meets the security criteria for its particular busi- 
ness sector. Members who pass initial vetting are certified into the program. Using 
a risk-based approach, CBP Supply Chain Security Specialists conduct on-site visits 
of foreign and domestic facilities to confirm that the security practices are in place 
and operational. 

C-TPAT is a voluntary public-private partnership program wherein CBP works 
with the trade community in adopting tighter security measures throughout their 
international supply chain. In return for making these enhancements members are 
afforded benefits to include reduced exams, dedicated cargo lanes (FAST), Stratified 
Exam, an assigned Supply Chain Security Specialist (SCSS). Potential members ini- 
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tially complete an online application. The SCSS then conducts internal record 
checks and evaluates the security profile provided. Upon completion of vetting the 
company then becomes a Certified member who will then be physically validated 
within a year of certification. 

In 2009, CBP continued to expand and strengthen the C-TPAT program and en- 
sure that certified member companies are securing their goods moving through the 
international supply chain to the United States. Teams of Supply Chain Security 
Specialists conducted validations and re-validations of C-TPAT members’ supply 
chains to ensure that security protocols are reliable, accurate, and effective. 

As C-TPAT has evolved, we have steadily increased both the rigor of the program 
and program membership. CBP has strengthened the C-TPAT program by clearly 
defining the minimum security requirements for all categories of participants wish- 
ing to take part in the program. As of Nov. 6, 2009, there were 9,509 companies 
certified into the C-TPAT program. CBP’s goal is to validate all partners within one 
year of certification, re-validate all companies not less than once every three years, 
and re-validate all U.S./Mexico highway carriers on an annual basis, due to the 
risks of compromise of trailers associated with the Southern Border Highway Car- 
rier sector of C-TPAT. 

C-TPAT’s 9,509 Certified Partners include 4,330 importers, 2,583 carriers, 821 
brokers, 784 consolidators/third party logistic providers, 56 Marine Port Authority 
and Terminal Operators and 935 foreign manufacturers. C-TPAT has conducted 
13,246 on-site validations of manufacturing and logistics facilities in 90 countries. 
301 C-TPAT importer partners have been designated Tier 3, meaning they have ex- 
ceeded the minimum security criteria and have been granted the highest level of 
program benefits. 

CONTAINER SECURITY INITIATIVE 

Because of the sheer volume of sea container traffic, containerized shipping is 
uniquely vulnerable to terrorist exploitation. To prevent terrorists and their weap- 
ons from entering the United States, CBP has also partnered with foreign govern- 
ments through our Container Security Initiative (CSI). CSI, which is the first pro- 
gram of its kind, was announced in January 2002 and is currently operational in 
58 foreign seaports — covering more than 80 percent of the maritime containerized 
cargo shipped to the United States. The program enables CBP to identify and in- 
spect high-risk cargo containers at foreign ports before they are shipped to our sea- 
ports and pose a threat to the United States and to global trade. CSI stations multi- 
disciplinary teams of CBP officers to work with host country counterparts to identify 
and examine containers that are determined to pose the highest risk for terrorist 
activity. 

CBP officers stationed at foreign CSI ports review 100 percent of the manifests 
originating and/or transiting those foreign ports for containers that are destined for 
the United States. In locations where the tremendous volume of bills prevents the 
CSI team at the port itself from performing 100 percent review, or during port shut- 
downs, CSI targeters at the National Targeting Center provide additional support 
to ensure that 100 percent review is accomplished. Utilizing the overseas CSI team 
and the CSI targeters at our National Targeting Center, CBP is able to achieve 100 
percent manifest review for the CSI program. In FY 2009, CBP officers stationed 
at CSI ports reviewed over nine million bills of lading and conducted over 56,000 
exams in conjunction with their host country counterparts. 

SECURE FREIGHT INITIATIVE 

The Secure Freight Initiative (SFI) is an effort to build upon existing port security 
measures by enhancing the U.S. Government’s ability to scan containers for nuclear 
and radiological materials in seaports worldwide and to better assess the risk of in- 
bound containers. SFI provides carriers of maritime containerized cargo greater con- 
fidence in the security of the shipment they are transporting, and increases the like- 
lihood of an uninterrupted and secure flow of commerce. This initiative is the cul- 
mination of our work with other U.S. Government agencies, foreign governments, 
the trade community, and vendors of leading-edge technology. 

CBP will prioritize future deployments of scanning systems to locations of stra- 
tegic importance by identifying seaports where non-intrusive imaging and radiation 
detection data would be most practical and effective in deterring the movement of 
weapons of mass destruction via containerized cargo. The additional scan data pro- 
vided by SFI will enhance DHS’ risk-based and layered approach to securing mari- 
time containerized cargo. We will continue to work with Congress to enhance the 
safety of our nation’s ports and the security of incoming cargo. 
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NON INTRUSIVE INSPECTION/RADIATION DETECTION TECHNOLOGY 

The deployment of imaging systems and radiation detection equipment has con- 
tributed tremendously to CBP’s progress in ensuring that supply chains bringing 
goods into the United States from around the world are secure against exploitation 
by terrorist groups. Non-Intrusive Inspection (Nil) technology serves as a force mul- 
tiplier that allows officers to detect possible anomalies between the contents of a 
container and the manifest. CBP relies heavily on the use of Nil, as it allows us 
to work smarter and more efficiently in recognizing potential threats. 

Prior to 9/11, not a single Radiation Portal Monitor (RPM), and only 64 large-scale 
Nil systems were deployed to our nation’s borders. By October 2002, CBP had de- 
ployed the first RPM at the Ambassador Bridge in Detroit. Today, CBP uses RPMs 
to scan 99 percent of all cargo arriving in the U.S. by land and sea. CBP, in partner- 
ship with the DHS Domestic Nuclear Detection Office (DNDO) and Pacific North- 
west National Laboratory (PNNL), has deployed 473 RPMs at northern border land 
ports of entry; 385 RPMs at southern border land ports of entry; 433 RPMs at sea- 
ports; and 55 RPMs at mail facilities. Currently, CBP has 232 large-scale Nil sys- 
tems deployed and 5,816 small-scale Nil units. Additionally, CBP has deployed 
1,515 Radiation Isotope Identifier Devices (RIIDs) and 19,365 Personal Radiation 
Detectors (PRDs). These devices allow CBP to examine 100 percent of all identified 
high-risk cargo. To date, CBP has used the deployed systems to conduct over 38 mil- 
lion examinations, resulting in over 8,300 narcotic seizures, with a total weight of 
over 2.6 million pounds, and over $28.6 million in undeclared currency seizures. 
Since RPM program inception in 2002, CBP has scanned over 404 million convey- 
ances for radiological contraband resulting in over 2.2 million alarms. CBP’s Labora- 
tories and Scientific Services spectroscopy group at the National Targeting Center 
has responded to some 21,599 requests from the field for technical assistance in re- 
solving alarms. To date 100 percent of all alarms have been successfully adjudicated 
as innocent. 

CBP is pleased to report that the final installation of RPMs along our shared 
northern border was commissioned on Oct. 29 at the Trout River, N.Y., port of 
entry. This milestone represents another critical step in the Department’s efforts to 
strengthen the interconnected U.S. border security network by deploying technology 
and personnel to key border locations to meet modern-day security needs. This re- 
cent milestone provides CBP with the ability to utilize radiation detection tech- 
nologies to scan 100 percent of trucks and personally owned vehicles arriving 
through both northern and southern border ports, and 98 percent of arriving sea 
containers. In addition, CBP officers now use hand-held radiation identification de- 
vices to scan 100 percent of private aircraft arriving in the U.S. from foreign des- 
tinations. 

The first generation RPM systems, although very sensitive, do have limitations. 
While they alert CBP officers to the presence of radiation, a secondary exam is nec- 
essary to identify the location and specific isotope causing the alert. In the event 
that a CBP officer is unable to positively resolve the alert, scientific reach back is 
available on a 24/7 basis through the National Targeting Center and CBP’s Labora- 
tory & Scientific Services Division located in the northern Virginia area. 

Understanding these limitations and the need for more precise radiological detec- 
tion architecture, DNDO was created in 2005 to focus on radiological and nuclear 
threats and develop new technologies that will improve the Nation’s ability to detect 
and identify radiological and nuclear weapons and material. One of these new tech- 
nologies is the next generation RPM, or the Advanced Spectroscopic Portal (ASP). 

The ASP is able to distinguish between actual threats and natural or medical ra- 
diation sources that are not security threats. In doing so, the ASP would enhance 
our detection capability, while significantly reducing the burden of responding to the 
numerous benign alarms that are mostly generated by everyday products. This 
would allow CBP to focus our staffing and resources on high-risk shipments and 
other border security initiatives. 

CBP COORDINATION WITH DNDO 

In our collaboration with DNDO, CBP brings knowledge of how our ports work, 
of the needs of our front-line officers, and of the operational requirements for new 
technologies that must work consistently in a broad array of environments. Addi- 
tionally, CBP is attuned to critical factors such as throughput and capacity as we 
seek to maintain an appropriate balance between security and the facilitation of 
cross-border travel and trade. 

CBP has worked closely with DNDO in the developmental and operational testing 
of the ASP. A complete, independent operational testing and evaluation will be con- 
ducted by the DHS Science and Technology Directorate’s Test and Evaluation and 
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Standards Division, along with the Director of Operational Test and Evaluation, 
when the system completes the current course of testing. CBP’s objective for oper- 
ational testing is to ensure that systems are suitable and effective in our operational 
environments. CBP provided DNDO with functional requirements for the ASP and 
has been actively engaged in every step of testing, including performance testing at 
the Nevada Test Site and integration testing currently ongoing at a mock port of 
entry at the Pacific Northwest National Laboratory. 

During CBP-led field validation and integration testing, CBP has been working 
closely with DNDO to assess each ASP system’s performance as an integrated unit, 
including reach back capability and ancillary equipment such as traffic lights and 
automated gate arms that are essential to maintaining positive control of vehicles 
at our ports of entry. In addition, CBP works with DNDO to assess and categorize 
each system’s issues to ascertain their technological impact on performance and 
their operational impact on front-line CBP officers — the users of the system. 

CBP will continue to work with DNDO toward certification by the Secretary, 
which is dependent on demonstrating a “significant increase in operational effective- 
ness” over existing first-generation radiation detection systems. At such time, the 
discussion could then turn to potential acquisition and deployment of ASP systems. 
Any decision to purchase and deploy ASPs in the operational arena will be based 
on mission needs, operational requirements, and a full understanding of mainte- 
nance and operational costs, to include a comprehensive cost-benefit analysis and 
an analysis of alternatives. 

CONCLUSION 

Technology plays an enormous role in securing the supply chain. Security tech- 
nology is continuously evolving, not only in terms of capability but also in terms of 
compatibility, standardization, and integration with information systems. It is im- 
portant to note that there is no single technological solution to supply chain secu- 
rity. As technology matures, it must be evaluated, and adjustments to operational 
plans must be made. Priority should be given to effective security solutions that 
complement and improve the business processes already in place, and which build 
a foundation for secure 21st century global trade. 

Mr. Chairman, Members of the Subcommittee, thank you again for this oppor- 
tunity to testify about CBP’s commitment to investing in new and emerging detec- 
tion technology, as well as some of the steps we have taken toward enhancing cargo 
security. I will be happy to answer any of your questions. 

Biography for Todd C. Owens 

Mr. Todd C. Owen was appointed to Executive Director of the Cargo and Convey- 
ance Security office, Office of Field Operations (OFO), in May 2006. Mr. Owen is 
responsible for all cargo security programs and policies for CBP, including the Con- 
tainer Security Initiative, the Customs-Trade Partnership Against Terrorism Pro- 
gram (C-TPAT) office, all non-intrusive inspection technology and radiation portal 
monitor deployments, the national canine enforcement program, the National Tar- 
geting Center-Cargo, and the 100 percent scanning initiative. Mr. Owen also coordi- 
nates U.S. Customs and Border Protection’s (CBP) maritime cargo enforcement poli- 
cies and activities with the U.S. Coast Guard and all air cargo efforts with the 
Transportation Security Administration. 

Mr. Owen previously served as the Director of the C-TPAT program from January 
2005 until May 2006. As Director, he strengthened management control and in- 
creased hiring, allowing for a significant enhancement in the level of foreign site as- 
sessments performed worldwide under this 9,000 member partnership program. 

Before arriving in Washington, D.C., Mr. Owen was the Area Port Director in 
New Orleans, where he was directly responsible for all CBP operations throughout 
Louisiana, managing 240 officers at seven port offices. Before being selected as Area 
Port Director, Mr. Owen spent eight years in South Florida where he held various 
trade related positions within OFO and the Office of Strategic Trade. 

Mr. Owen began his career with U.S. Customs Service in 1990 as an Import Spe- 
cialist in Cleveland, Ohio. 

Mr. Owen is a career member of the Senior Executive Service and was a senior 
executive fellow at Harvard University’s John F. Kennedy School of Government. 

He is a graduate of John Carroll University in Cleveland, Ohio, and holds a Mas- 
ter’s degree in Public Administration from St. Thomas University in Miami, Florida. 

Chairman Miller. Thank you, Mr. Owen. Dr. Hagan for five 
minutes. 
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STATEMENT OF DR. WILLIAM K. HAGAN, ACTING DEPUTY DI- 
RECTOR, DOMESTIC NUCLEAR DETECTION OFFICE, DE- 
PARTMENT OF HOMELAND SECURITY 

Dr. Hagan. I am sorry. I am losing my voice. Good afternoon 
Chairman Miller, Ranking Member Broun and the distinguished 
Members of the Subcommittee. As Acting Deputy Director of the 
Domestic Nuclear Detection Office, DNDO, at the Department of 
Homeland Security, I am honored to be here with the U.S. Customs 
and Border Protection Executive Director for Cargo and Convey- 
ance Security, Mr. Todd Owen. 

I would like to thank the Committee for the opportunity to pro- 
vide this update on our Advanced Spectroscopic Portal Program, or 
ASP. Since our last hearing, ASP systems have undergone another 
round of field validation conducted by CBP. The purpose of field 
validation is to produce information about the operational charac- 
teristics of the system and correct any issues that must be resolved 
before moving onto the last test event for the system, the Oper- 
ational Test and Evaluation (OT&E). 

The July field validation identified several issues. We worked 
with CBP to classify the severity of these operational and technical 
issues and DNDO has since addressed all the issues initially iden- 
tified. Because the systems are configuration controlled, adjust- 
ments were checked and validated through use of a replay tool and 
regression testing to ensure that sensitivity to threats was not de- 
creased below guidance. Although a few issues remain to be re- 
solved, we are optimistic that field validation can be restarted in 
the near future. 

After CBP completes field validation, OT&E will be conducted by 
the DHS Science and Technology Directorate’s operational testing 
authority. The current path to certification includes field validation 
and OT&E, as well as preparation of all the departmental acquisi- 
tion program requirements, including presentation of the program 
before the DHS Acquisition Review Board, or ARB, for a production 
decision. We are committed to following the Department’s Manage- 
ment Directive 102-01. As such, we cannot speculate about the out- 
come of the ARB decision. The Secretary’s decision regarding cer- 
tification will follow the ARB. 

Beyond testing of the ASP systems, deployment of our current, 
low-rate initial production ASP units is now being considered. The 
fiscal year 2010 appropriations bill signed three weeks ago contains 
specific language encouraging incremental deployments of ASP, as 
did the report from the National Academy of Sciences, and we are 
planning discussions with CBP regarding the best way to deploy 
these units once OT&E is completed. 

However, one new obstacle has emerged that will greatly impact 
the path forward. The United States is facing a severe helium-3 
shortage. Helium-3 is a gas that is used for neutron detection with- 
in both current and next-generation RPMs. Because this decreased 
supply affects multiple agencies within the Federal Government 
and beyond, the White House has convened an Interagency Policy 
Committee, including DHS, to discuss the issue and possible solu- 
tions. This group decided in September that no more helium-3 will 
be allocated to RPMs for the time being. We are currently leading 
interagency efforts to identify alternative neutron detectors, and we 
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are working to assess the impact this will have on our deployment 
strategy and path forward. For now, we plan to continue field vali- 
dation and operational testing evaluation in exploring our options 
for the path forward beyond that. 

Additionally, I have also been asked to address the status of im- 
provements to the currently deployed RPM systems, the PVT sys- 
tems. Some have asked if software or algorithm improvements in- 
cluding improvements to the energy windowing algorithms that are 
currently deployed could provide enhanced capabilities for PVT sys- 
tems. We are currently funding work at Pacific Northwest National 
Laboratory and the Johns Hopkins University Applied Physics Lab- 
oratory to improve energy windowing and conduct related research 
to determine what kinds of PVT improvements may be possible. 
This research is still ongoing. That being said, it is important to 
note that any improvements to PVT detectors will only improve 
primary inspection capabilities or detection. Current performance 
limitations, and secondary inspection where CBP officers require 
the ability to identify radiological sources, are not addressed by im- 
provements to PVT technology. Better energy windowing algo- 
rithms will not change this fact. 

As a final note, I would like to emphasize that the ASP program 
is only one piece of the multi-layered solution we have termed the 
global nuclear detection architecture. This strategy calls for the use 
of multiple preventive actions at our ports as well as security 
measures along all potential pathways beyond, at and within the 
Nation’s borders. 

We will continue to work with our partners within DHS, other 
federal departments, State and local agencies and the Members of 
this subcommittee and the Congress to keep the Nation safe from 
radiological and nuclear terrorism. 

This concludes my statement. I thank you for your attention, and 
I will do my best to answer any questions you may have. 

[The prepared statement of Dr. Hagan follows:] 

Prepared Statement of William K. Hagan 


Introduction: 

Good afternoon Chairman Miller, Ranking Member Broun, and distinguished 
Members of the Subcommittee. As Acting Deputy Director of the Domestic Nuclear 
Detection Office (DNDO) at the Department of Homeland Security (DHS), I would 
like to thank the Committee for the opportunity to provide a status update on the 
Advanced Spectroscopic Portal (ASP) program. 1 would also like to thank the Com- 
mittee for its support of DNDO’s mission to reduce the risk of radiological and nu- 
clear terrorism to the Nation. 

In late June of this year, I provided testimony about our next-generation radiation 
portal monitors (RPMs), including where RPMs are deployed and how DHS’ U.S. 
Customs and Border Protection (CBP) operates this technology at our ports of entry 
(POEs). My testimony today will include a status update and the path forward for 
the ASP program. I will also describe operations during field validation and im- 
provements for poly-vinyl toluene (PVT) systems. 

Energy Windowing for PVT Systems: 

Before I get into an update on the ASP Program, I have been asked to address 
the status of improvements to PVT systems. For some time, there have been various 
questions about the possibility of exploring energy windowing improvements to the 
currently deployed PVT systems. Energy windowing is an algorithmic alarm method 
that can be applied to plastic scintillator-based RPM systems to improve operational 
sensitivity to certain threat sources while reducing the alarm rates from naturally 
occurring radioactive material. Some have asked if software or algorithm improve- 
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merits could provide enhanced capabilities for PVT systems that would achieve per- 
formance similar to that of the ASP systems. Algorithms that provide energy 
windowing for PVT systems were introduced into the currently deployed systems in 
2007, and DNDO is currently funding work both at the Johns Hopkins University 
Applied Physics Laboratory (APL) and at Pacific Northwest National Laboratory 
(PNNL) to determine if further gains can be made through additional energy 
windowing techniques. We have also asked that, wherever possible, the labs work 
cooperatively on these efforts. Funding for the work at PNNL is approximately $1.6 
million of Fiscal Year (FY) 2008 appropriations, and investments in APL for studies 
related to energy windowing are approximately $90,000 of FY 2009 appropriations. 
Scientists are thus far uncertain whether additional gains in operational perform- 
ance can be coaxed from PVT RPMs by using more energy windowing, because the 
limits of passive detection with plastic-based detectors are coupled with a need to 
keep false alarms at a minimum. As new techniques are studied and evaluated, it 
is important to note that the scintillation properties of PVT detectors are fundamen- 
tally different than the ASP technology and we continue to evaluate the operational 
effectiveness of next-generation systems concurrently. 

Current Status of the ASP Program: 

DNDO continues to enhance the capability to detect and report attempts to im- 
port, possess, store, develop, or transport nuclear or radiological material for use 
against the Nation. Part of our work involves working with DHS partners to provide 
the equipment and systems they need to perform their missions, including scanning 
cargo for possible radiological or nuclear threats. After 9/11, considerable concern 
was raised about the possibility that terrorists could use the enormous volume of 
cargo flowing into the United States as a means to bring in nuclear material or a 
nuclear weapon. By far, the largest mode for incoming cargo is maritime shipping 
containers, with approximately 11 million containers coming into the country every 
year. Additionally, in the Security and Accountability for Every (SAFE) Port Act of 
2006, Congress mandated that all containers coming in through the 22 top volume 
ports be scanned for radiation by the end of 2007. Thus, considerable effort and re- 
sources have been devoted to this mode of transportation to provide comprehensive 
radiological and nuclear detection capabilities, particularly at POEs. 

CBP currently scans cargo entering at our nation’s POEs using PVT-based radi- 
ation portal monitors (RPMs) that can detect radiation, but cannot distinguish be- 
tween threat materials and naturally occurring radioactive materials (NORM), such 
as kitty litter and ceramic tiles. Narrowing down alarms to just those for dangerous 
materials is especially important for POEs that have a high volume of containers, 
or those that see a high rate of NORM. 

Building on previous work within CBP and the DHS Science and Technology Di- 
rectorate (S&T), DNDO initiated the ASP program in 2006 to develop next-genera- 
tion technology that can both detect radiological material and distinguish between 
threat and non-threat materials. ASP systems have shown significantly improved 
capability to distinguish radiological threats from non-threats over the hand-held in- 
struments currently used in secondary screening. Thus, the introduction of ASP sys- 
tems is expected to not only reduce the number of unnecessary referrals and false 
positives in primary scanning but increase the probability of detecting dangerous 
materials in secondary. I want to be clear here that ASP cannot and will not be 
acquired and deployed until an Acquisition Review Board (ARB) has been conducted 
and the Secretary of Homeland Security has certified that the technology provides 
a significant improvement in operational effectiveness over current systems. Al- 
though we are getting closer to these decision points, testing and evaluation still re- 
mains so that all the data required to make informed decisions has been accumu- 
lated, analyzed, and documented. 

To date, two ASP vendors have developed systems that have completed the fol- 
lowing 2008-2009 tests: System Qualification Testing, designed to demonstrate that 
ASP units are manufactured in accordance with processes and controls that meet 
the specified design requirements; and Performance Testing at the Nevada Test Site 
(NTS), designed to evaluate ASP, PVT, and radioisotope identification devices 
(RIID) detection and identification performance against controlled, realistic threat 
materials, shielding, and masking scenarios. One vendor has also completed Inte- 
gration Testing, designed to determine whether the ASP systems are capable of op- 
erating and interfacing with the other equipment found in operational settings. This 
vendor has now begun field validation testing, designed to exercise ASP systems in 
a stream of commerce environment at POEs. 

The second vendor has experienced technical and accounting issues that have 
caused its testing schedule to lag behind. DNDO is in the process of determining 
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the best path forward with this vendor. The remainder of my testimony will focus 
solely on the first vendor. 

Field Validation: 

Since our last hearing, CBP conducted an additional round of field validation at 
operational POEs. You may recall that CBP operated the ASP systems at four field 
validation sites in January and February of 2009 for a period of two weeks. During 
this time, the systems were run in tandem with the PVT systems to scan incoming 
cargo conveyances, and were able to collect data in an operational port environment 
with real flow-of-commerce. During these operations, the ASP systems showed high- 
er than expected alarm rates for three industrial sources. 

Following this first round of field validation, the ASP thresholds were adjusted 
to more effectively eliminate alarms on benign sources and were retested to ensure 
that the sensitivity for detection of special nuclear material and threats did not fall 
below guidance requirements. Utilizing a computer-based replay tool, we were able 
to determine that the adjustments solved the problem without decreasing the prob- 
ability that the system will detect and identify threats. Additionally, the systems 
were put through regression testing to determine if the problem had been appro- 
priately addressed. 

After the threshold adjustments were fine-tuned for the three specific isotopes 
that were problematic in the winter, CBP restarted field validation in July. This 
testing identified two additional issues. First, there was a single mission-critical 
fault at one field validation site in which the software database system failed to 
scan multiple conveyances, and did not immediately notify the CBP Officer of the 
problem. There was no security threat at this port because ASP systems were oper- 
ating in parallel with the current generation systems. A technical representative 
from the vendor was summoned and the issue was rectified. The cause of the prob- 
lem has since been identified and fixed. To assure that the fix did not create any 
unintended problems, the software has been successfully regression tested. 

The second technical issue identified during July field validations was a higher 
than expected rate of false alarms for certain special nuclear material (SNM) 
threats. Since ending this second round of testing, the ASP thresholds were adjusted 
to eliminate this problem, and the systems were retested to ensure that the sensi- 
tivity for detection of special nuclear material and threats was not reduced below 
guidance requirements. Use of the replay tool and regression testing validated that 
the false alarm rate will be reduced below guidance requirements, and it is antici- 
pated that another round of tandem operation field validation will verify the effec- 
tiveness of the settings. In this case, the threshold adjustments resulted in an ac- 
ceptable reduction in the sensitivity of the system, while remaining more sensitive 
than required by the original specifications. 

The fact that the SNM false alarms did not occur at the same rate in the first 
round of field validation as seen in the second round is a good example of the need 
for robust, multi-stage testing, since it may have revealed a sensitivity to seasonal 
changes and cargo contents that was not expected. Such oddities are typical when 
a new system transitions from the lab to the real world, and we will continue to 
learn and improve as testing progresses. At present, DNDO and CBP are in discus- 
sions to establish the ground rules for starting the next period of tandem field vali- 
dation evaluations. Following the completion of tandem operations, ASP systems 
will undergo solo field validation evaluations. 

Path Forward: 

Once all the phases of field validation are completed to the satisfaction of both 
DNDO and CBP, ASP systems will complete independent operational testing and 
evaluation (OT&E) conducted by the S&T Operational Testing Authority. Test data 
will be analyzed and provided to inform the Secretary’s certification decision. DNDO 
is also engaged with the National Academy of Sciences (NAS) to perform an addi- 
tional review of ASP testing and inform the certification process, as required in the 
FY 2008, 2009, and 2010 Homeland Security Appropriations bills. 

Additionally, DNDO is working to develop a cost-benefit analysis that will analyze 
the cost effectiveness of deploying ASP systems in several different configurations. 
This cost-benefit analysis methodology is still being processed with the available 
data and requires additional data from the remaining tests before it can undergo 
review within the Department and be finalized. 

The current path to certification includes testing as described above, accompanied 
by the analysis of results, to ensure that Secretary Napolitano has sufficient infor- 
mation and all the Departmental acquisition program requirements are met. ASP 
systems have been under review and evaluation for over three years now, but we 
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are focused on informing a decision to go forward with acquisition and deployment 
with the appropriate processes. Such a decision will be made only when it has been 
determined that ASP will increase the probability of detecting dangerous materials 
while minimizing operational burdens, rather than based on a pre-determined 
timeline. 

The changes and continued diligence that DNDO exercises in conjunction with the 
ASP program will ensure that any eventual certification and acquisition decisions 
are consistent with DHS priorities and made with a documented acquisition man- 
agement foundation. The ASP program will also be presented to the DHS ARB for 
an MD 102-01 milestone decision for purchase and deployment of ASP. This is a 
very well-defined milestone that was developed for all large programs within the 
Department of Homeland Security. Items such as mission needs, operational re- 
quirements, analysis of alternatives, etc., are part of the MD 102-01 process — no 
production units can be purchased and deployed without successfully navigating the 
process. To be clear, the Secretarial certification requirement is in addition to the 
MD 102-01 deployment decision. DNDO intends to present both the Secretary and 
the ARB with all the necessary information to make decisions about the operational 
effectiveness and potential optimal deployment of ASP systems. 

Conclusion: 

DNDO will continue to work with CBP and other partners within and beyond 
DHS to improve the Nation’s ability to detect radiological and nuclear threats at our 
ports and borders. DHS is facing a challenge; we must balance facilitating the flow 
of commerce at our ports and borders with the need to sufficiently scan cargo for 
radiological or nuclear threats before it enters our Nation. As both the President 
and Secretary have said, the Nation needs more technology to meet its security 
challenges and the technologies that DNDO is pursuing, of which ASP is but one 
example, are a critical component in addressing that challenge. 

Our efforts to develop and evaluate ASP systems are based on sound, proven test- 
ing and evaluation processes used with proven success across government and aca- 
demia. Current test results are capturing the benefits of ASP systems, and the re- 
views to date have provided a valuable assessment of the program and identified 
a number of key lessons learned. As we collect more operational data for the ASP 
systems, we are better able to determine the optimized settings for detecting and 
identifying threats, while facilitating the flow of legitimate commerce. 

I welcome and appreciate the Committee’s active engagement with this program, 
and look forward to continuing our cooperation as we move forward together. Chair- 
man Miller, Ranking Member Broun, and Members of the Subcommittee, I thank 
you for your attention and will be happy to answer any questions that you may 
have. 


Biography for William K. Hagan 

Dr. William Hagan serves as the Acting Deputy Director for the Department of 
Homeland Security’s (DHS) Domestic Nuclear Detection Office (DNDO). Prior to 
January 2009, Dr. Hagan served as the Assistant Director of the Transformational 
Research and Development (R&D) within DNDO. Dr. Hagan was responsible for 
long-term R&D, seeking technologies that can make a significant or dramatic posi- 
tive impact on the performance, cost, or operational burden of detection components 
and systems. 

Before his work with DNDO, he was a Deputy Business Unit Manager at Science 
Applications International Corporation (SAIC). Business areas included nuclear 
technology (analysis, detection, and applications), telecommunications, optics, trans- 
portation, system integration, and technology assessments during his thirty years 
at SAIC. 

Dr. Hagan earned a Bachelor of Science in Engineering Physics in 1974, Master 
of Science in Physics in 1975, and Master of Science in Nuclear Engineering in 1977 
from the University of Illinois at Urbana. He received his Ph.D. in Physics from the 
University of California-San Diego in 1986. He holds three patents. Dr. Hagan was 
appointed to Senior Executive Service in 2006. 


Discussion 

Chairman Miller. Thank you, Dr. Hagan. I now recognize my- 
self for five minutes. 
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CBP Procedures After a Primary Alarm and the 
Effect of False Positives 

Mr. Owen, I hope that CBP’s procedure is, if you identify nuclear 
materials, the kind of nuclear material that would be used in a nu- 
clear weapon or for a dirty bomb, that your procedures would re- 
quire that you act like your hair was on fire, that you act with 
great urgency. Without getting into anything that is classified in 
this setting, could you describe what your procedures are? 

Mr. Owen. Yes, sir. Whenever we have any alarm of a PVT on 
primary, the cargo or the passenger vehicle is then sent into sec- 
ondary because again, the PVTs at this point only detect radiation, 
they do not identify what it is. 

In the secondary, we use the hand-held Radiation Isotope Identi- 
fication Device. If that determination is that we have special nu- 
clear materials, we then implement what we call SIN procedures: 
secure, isolate, notify. We secure the area, we isolate the convey- 
ance, and then we begin our notifications back to our Laboratory 
and Scientific Services folks at our National Targeting Center for 
cargo here in Northern Virginia. The CBP officers would transmit 
the radiation spectra to the Laboratory and Scientific Services peo- 
ple to make a determination as to what exactly what we have 
there. So as this is going on, again, we are in secondary. It is a 
more controlled arena. Depending on the footprint of the port, de- 
pending on the layout, the time of day, the traffic, will impact how 
much of a disruption it will be to some port operations. But clearly, 
whenever we have a higher level threat such as the special nuclear 
materials, we do act accordingly. There is some disruption to the 
port operations until we resolve that alarm. 

Chairman Miller. You said secure, secure and identify, secure 
the area. How big an area do you secure? 

Mr. Owen. Mostly it is just in the secondary area with what we 
have. We will create a perimeter of secure area if you will. It will 
not require an automatic shutdown of neighboring activities or any- 
thing like that. As we do the initial readings, send it back to our 
Laboratory and Scientific Services folks — clearly, if the issue has 
escalated, if it is of grave concern to us, your exclusion zone would 
expand and the level of activity would increase. 

Chairman Miller. Okay. Do you have any idea how many hours 
ports like New York, New Jersey, Los Angeles, Long Beach, would 
be closed or how they would be affected by false positives for spe- 
cial nuclear materials? 

Mr. Owen. I don’t have any data as to how many times this has 
happened or was shut down, but those are the procedure. And real- 
ly again, based on the layout of the port, if it is a large seaport, 
you can have just one terminal affected by an alert of special nu- 
clear materials through one of our detection devices. If it is a 
smaller land border crossing, you could theoretically have a much 
greater impact on that type of operation. So there is an impact de- 
pending on the different variables as to how much of an impact, 
but clearly there is an impact whenever we have an alert of a spe- 
cial nuclear material. 

Chairman Miller. Okay. Well, obviously one of the most imme- 
diate problems with false positives is disruption of commerce, dis- 
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ruption of ports and borders. But I remember from college that the 
first time the fire alarm went off in the dorm, everyone rolled right 
out of bed and went outside, but by the time it happened three or 
four times because of a prank, all anyone did was just put a pillow 
over their head to make the noise — so they could sleep through the 
noise. I don’t ever want it to become routine if we have an alert 
that there is a special nuclear material. Obviously, if there is a real 
fire, the fire alarm would have done no good at all. 

Mr. Aloise, the July round of field validation tests, there were, 
you said, false positives. The machinery, the device, ASPs detected 
materials that were not there at all. Can you give us the details 
of how many false positives or was it one or two machines, was it 
all the machines? What happened? 

Mr. Aloise. Yes, I believe, and CBP can confirm this, there was 
four per thousand at the smaller ports and eight per thousand, 
eight false positives per thousand conveyances at the larger ports 
during the testing in July. 

Chairman Miller. And was that across all machines or was that 
just one or two machines that seemed not to be quite right? 

Mr. Aloise. I believe it was across all machines, yes. 

Chairman Miller. So it was a problem with technology generally 
not with a problem with a specific 

Mr. Aloise. With any one machine. 

Chairman Miller. Okay. My time has almost expired, so I will 
recognize Dr. Broun for five minutes. 

Mr. Broun. Thank you, Chairman. I still am going to be very 
generous with time. You have been very generous with me, and I 
will continue to be generous with you at any point. I want to make 
the Chairman 

Chairman Miller. After serving for two years with Dana Rohr- 
abacher as a Member of the Subcommittee, it is really not a prob- 
lem. 


Steps Taken to Reduce False Positives and Negatives 

Mr. Broun. Thank you, Mr. Chairman. Some of the field valida- 
tion tests reveal high levels of false positives for the special nuclear 
materials. This is a serious problem because any identification of 
any special nuclear materials requires a robust response from CBP 
officers. What is DNDO doing to address this issue? And you might 
also — I am concerned not only about false positives but false nega- 
tives. And I really haven’t heard anything about that so if you 
could deal with that, too? 

Dr. Hagan. Regarding what we are doing about the false 
positives, the field validation showed a large number or unaccept- 
ably high number of calls for SNM (Special Nuclear Material). We 
have adjusted the threshold levels for the ASP and run a replay 
tool to demonstrate that in fact the number of false alarms, if these 
threshold settings had been used instead of what was used in July, 
that almost all of those false alarms would have been eliminated. 
The number would have dropped from about, according to my data 
here, from about 49 to seven. Now, no system is flawless or perfect. 
And so it is not likely that we will ever build a system that will 
produce no false alarms in primary. But the levels that we are op- 
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erating now according to the replay tool are within the accept- 
ability range. 

Regarding false negatives, in the flow of real commerce in the 
field validation activity, there is really no way to know if you have 
dismissed something that you shouldn’t have. Now, we have testing 
at another place, the Nevada Test Site, where we tested that, and 
I don’t recall what the false negative rate was, but it was quite low. 

Mr. Broun. As a physician, I am concerned about, and in 
science, as you know, talk about specificity and sensitivity. Several 
independent sources have already indicated that ASP only margin- 
ally increases our ability to detect lightly shielded special nuclear 
materials. If we manipulate those thresholds, won’t this decrease 
that margin even more, and if this is the case, how would that af- 
fect any future cost benefit analysis? 

Dr. Hagan. The change in — when the threshold settings were 
changed, again using the replay tool, we were able to calculate 
what the difference or see what the difference was in the sensi- 
tivity. It is a reduced sensitivity as with any detector. When you 
make your system — when you reduce, make changes to reduce the 
false alarm rate, the sensitivity is usually affected. So this is true 
of this detector as well. But when we compare the sensitivity of the 
system before the threshold settings were made to that after, while 
it was reduced, the reduction was rather slight and the perform- 
ance of the system is still within the specification and the guid- 
ance. 

Mr. Broun. In order to verify and validate the threshold changes 
made to ASP systems, a replay tool was developed. Has this tool 
been validated by an independent party? 

Dr. Hagan. Yes, it has. 

Mr. Broun. Okay. And who is that party? 

Dr. Hagan. The Johns Hopkins APL, Applied Physics Lab. 

Mr. Broun. And what was the results of that? 

Dr. Hagan. Okay. Well, maybe the best way to convey that is to 
comment or to give you some of the results from the replay tool val- 
idation exercise. So when we say a replay tool, we mean we want 
a tool that when given the same input data as the real system, it 
produces the same results as the real system. And so the replay 
tool was given the real data for 11,000 occupancies, and the results 
were compared to what actually came out of the system. The re- 
sults were identical for all 11,000 except for eight. Those eight that 
were found, the reason that the differences were slightly different, 
the results were slightly different, is that the codes were run on 
two different computers, and there was a slight round-off error dif- 
ference between the two computers. And because of that, there 
were eight occupancies that yielded slightly different results. Other 
than that, the results were identical. 

Mr. Broun. Thank you. Mr. Aloise or Dr. Persons, do you all 
have a response to that? 

Mr. Aloise. We would say this and then I will let Dr. Persons 
answer as well. As you recall in June when we testified, we said 
there was a marginal improvement in the Nevada testing over 
what the PVTs were, and that marginal improvement, because of 
this reducing of the thresholds has now gotten smaller. So when 
they do the cost benefit analysis, all this has to be factored into 
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that cost benefit analysis because now the ASP seemingly does not 
have the performance that it initially did when we testified here 
earlier this year. So that is one point. 

The other point is we have not looked at the replay tool. We have 
not talked to Johns Hopkins, so we can’t really comment on the re- 
play tool. Dr. Persons, would you like to 

Dr. Persons. Well, just to confirm what Mr. Aloise was saying, 
we have not been able to look at that, but one of our key points 
on this is just the idea of an independent verification or validation 
of such a tool to do that. And in interviews with CBP officials, for 
example, there was a concern about perhaps having an institution 
such as the American National Standards Institute, or ANSI, do 
something more standardized oriented body to do that. There is 
also of course when you are dealing with software development 
issues which are implied in these systems with this level of com- 
plication, then you would look at of course the software models, the 
software standards, how those were constructed, how they execute 
the algorithms and perform under test. 

Mr. Broun. Dr. Hagan, I would like for you to, if you would, sub- 
mit that report for the record for us 

Dr. Hagan. Sure, in the form of a letter. 

Mr. Broun. — and a discussion about it. Okay. Good. Thank you, 
sir. Mr. Chairman, I yield back. 

Chairman Miller. Thank you, Dr. Broun. The Chair joins Dr. 
Broun in that request. 

Ms. Dahlkemper for five minutes. 

Mission Critical Failure 

Ms. Dahlkemper. Thank you, Mr. Chairman. Thank you to all 
those here testifying today. I want to go to one of the most basic 
problems and I guess one of the most alarming was the mission 
critical failure when two dozen cargo trucks were permitted to go 
through the ASP in order to be screened and actually were not op- 
erating at that time. And supposedly this has been rectified by the 
contractor with an indicator light. Is that correct? 

Dr. Hagan. No. 

Ms. Dahlkemper. Nothing has been rectified? 

Dr. Hagan. It has been rectified but not with an indicator light. 
There is a message that gets shown to the operator on screen that 
says that the system is in mission critical failure and should not 
be operated. 

Ms. Dahlkemper. So prior to this there was no way to detect if 
there was 

Dr. Hagan. There was a message but it was confusing and not 
clear, and so that is what led to the incident. By the way, I should 
comment that this only happened one time in one system in all the 
testing in validation runs that have been going on for approxi- 
mately for a year. So we don’t think it is going to happen again, 
but we modified the software to properly notify the operator. 

Ms. Dahlkemper. I can buy a $15 smoke detector for my house 
and it will tell me when it is beginning to not work, and when it 
doesn’t work, it will beep so I have to get up in the middle of the 
night and take the battery out. But anyway, I just wanted to make 
sure that that problem is rectified at this point. 
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Dr. Hagan. It has been rectified, it has been regression tested, 
and we are confident that the message will be properly displayed. 
We are struggling a bit with this one error or issue that has come 
up because it has only happened once. So we don’t think it will 
happen again. But that is the point of field validation, is to shake 
out some of the issues and problems before you move on to the ac- 
tual operational test. 

Ms. Dahlkemper. How may of these are in operation in this field 
testing? How many? 

Dr. Hagan. Six. 

Ms. Dahlkemper. Six? And how many of the PVTs do we have 
currently? 

Mr. Owen. Just under 1,400 deployed nationwide at the land 
borders and seaports and some mail facilities. 

Energy Windowing to Improve PVT Performance 

Ms. Dahlkemper. Okay. One of the things I wanted to ask, Mr. 
Owen, is on the GAO’s previous work on the ASPs, they found that 
the ASPs provide a marginal improvement of the PVTs to attack 
the certain types of radioactive sources under a handful of very 
specific scenarios. But the GAO has also recommended that DNDO 
invest in ways to increase performance of the existing radiation 
monitors known as PVTs by a software improvement process called 
energy windowing. Do you believe that more efforts should be made 
to investigate the benefits of the energy windowing? 

Mr. Owen. Yes, our position remains that we need to fully maxi- 
mize what the detection capabilities of the existing technology of 
the PVTs is. So we believe there are gains that have already been 
made with energy windowing, and we would rely on our scientific 
partners to tell us if we have reached the edge of that envelope as 
to how much they can do, either with the existing algorithms or 
with advanced algorithms that may be there. So as the operator 
and end-user of the equipment, yes, we would like to etch out as 
much detection capabilities as we can from the PVTs, recognizing 
that even with energy windowing, there will still not be an identi- 
fication device, just a detection device. 

Ms. Dahlkemper. Mr. Aloise, how much has been spent on en- 
ergy windowing work by DNDO or DHS in the last few years com- 
pared to how much has been spent on developing ASPs? 

Mr. Aloise. I don’t have that information, but I do know that we 
have been told that further research has sort of been slow-balled, 
and we would have liked to have seen them a lot further by now 
in where the energy windowing is with the PVTs. 

Ms. Dahlkemper. Could I get that information? Could that infor- 
mation be found, Mr. Owen, in terms of how much work has been 
done? 

Dr. Hagan. Yeah, I can give you the numbers right now. 

Ms. Dahlkemper. Okay. 

Dr. Hagan. For energy windowing, we have funded a total of 
about $1.9 million again to Pacific Northwest National Laboratory 
and the Johns Hopkins University Applied Physics Laboratory. 

Ms. Dahlkemper. And how does that compare to how much has 
been spent on ASPs? 
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Dr. Hagan. It is a lot less. I don’t know the exact ratios or any- 
thing. 

Ms. Dahlkemper. What is the total? You don’t know what the 
total has been spent? 

Dr. Hagan. Well, I think the number that was quoted I wouldn’t 
argue with. I think it was $230 million over the last 

Ms. Dahlkemper. It is $1.9 million over the last how many 
years? 

Dr. Hagan. Which 

Ms. Dahlkemper. Last few years. When did we start? I am new 
to Congress so when did we start working on ASPs. 

Dr. Hagan. Oh, ASPs? 2004 or 2005. I actually don’t know. I 
wasn’t here then. 

Ms. Dahlkemper. So we have spent $230 million in that time pe- 
riod on that? 

Dr. Hagan. Right. 

Ms. Dahlkemper. And the energy windowing, you started work- 
ing on that? 

Dr. Hagan. That was in fiscal year 2009. I think the actual work 
started in July of ’09 at PNNL. 

Ms. Dahlkemper. So since just this past July we spent $1.9 mil- 
lion? 

Dr. Hagan. Oh, we haven’t spent it. We have obligated it. 

Ms. Dahlkemper. Obligated it? 

Dr. Hagan. I don’t think it has been actually expended yet, but 
the work is ongoing. 

Ms. Dahlkemper. Okay. And that just started this year, in July? 

Dr. Hagan. Yes, in July, right. 

Ms. Dahlkemper. My time is expired. I yield back. 

Chairman Miller. Thank you, Ms. Dahlkemper. Mr. Bilbray for 
five minutes. 

Mr. Bilbray. Thank you, Mr. Chairman. I apologize for being 
late, and so if my questions have already been answered, you can 
remind me my tardiness was inappropriate. 

The CBP Inspection System 

Mr. Owen, the issue of inspection, do we have a formal, multi- 
tier inspection system and when we talk about the inspection is 
there any pre-entrance inspection before they get to the port facili- 
ties? 

Mr. Owen. We do not pre-inspect cargo. What we do have is part 
of our layered cargo security strategy. We receive advanced infor- 
mation on all cargo and all modes before it arrives in the United 
States, different timelines, whether it is coming by air, by truck, 
by rail or maritime. With that advanced information, we make an 
initial assessment if the cargo poses a risk. In our maritime envi- 
ronment through our container security initiative, we have officers 
deployed in 32 countries at 58 seaports that can then conduct an 
inspection on the highest risk maritime cargo before it is put on a 
vessel coming to the United States. We do not have a similar pro- 
gram in air or in land. So even though we have that advanced in- 
formation before the truck or the train or the airplane arrives in 
the United States, we can identify the highest risk, but then it 
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would be looked at at the port of arrival. So we do not have pre- 
inspection overseas for cargo. 

Mr. Bilbray. Okay. So we do have it for the maritime. Do we 
have any structure at all that if there was the ability to basically 
modify the arrival system, in other words hold them offshore until 
we could board the craft and be able to inspect it? 

Mr. Owen. Yes, sir. Absolutely. We work very closely with the 
Coast Guard. If there are any threats on a vessel, we will keep the 
vessel out at sea to address those threats or concerns with the crew 
or whatever the case may be. Likewise with the air cargo, we re- 
ceive the information four hours before the plane lands, so in most 
cases if we have a concern that aircraft can be diverted to have ad- 
dress that threat, yes, sir. 

Mr. Bilbray. And I am glad to hear that because, let us face it, 
the port facilities tend to be much more centrally located in high 
population areas than either the port of entries for vehicles or with 
the aircraft, so thats preemptive. 

The Helium-3 Shortage and Potential Alternative 
Materials 

The issue of helium-3, as we are fighting to try to make what we 
have stretch, any discussion of the crisis and what we would need 
to do to crank up the breeder so we can produce more and make 
it available for you? 

Dr. Hagan. There has been a lot of discussion. This issue has 
really risen quickly to the top of a lot of people’s lists, including the 
White House. They have formed an Interagency Policy Committee 
to address this that involves not only DHS but of course the White 
House folks, DOE, DOD and so forth. The problem is rather severe. 
The demand will probably outstrip the supply by a factor of ten. 
There is just not very much. And there are other uses for this ma- 
terial, for helium-3 that involve medical uses and so forth that 
frankly will probably end up having higher priority, although I 
can’t say that for sure. 

Mr. Bilbray. So the fact that this is a strategic supply problem 
that we need to address as a nation? 

Dr. Hagan. Yes. I believe that is the case. Now, there are — so for 
some applications, like low-temperature physics research, helium- 
3 is pretty much the only — you are almost researching the material 
itself, so it can’t be replaced. 

For our applications in neutron detection, there are other poten- 
tial alternatives which we are, DNDO, is taking the lead for the 
interagency group to explore. We have been funding research in 
this area for several years already. So we are looking at alternative 
means of detecting neutrons. But because we don’t know what 
those means are today, there are several, as I say, potential alter- 
natives. But it can still take a year or two probably to sort of test 
those, evaluate them and make sure that the designs work. So it 
is a real problem for us. That is why it has thrown a real glitch 
into our planning process. 

Mr. Bilbray. Are there any other jurisdictions or agencies 
around the world that are specifically looking at developing this ca- 
pability? 

Dr. Hagan. To make helium-3? 
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Mr. Bilbray. Yeah. 

Dr. Hagan. Well, there are places 

Mr. Bilbray. We don’t have any others domestically. Is there 
anybody domestically looking at it? 

Dr. Hagan. There is some thought to perhaps going into natural 
gas wells and separating some of the helium from that and then 
distilling that, pulling out the helium-3 from that. There is a possi- 
bility of a source in — well, several sources in other countries. I 
probably shouldn’t say much more than that. 

Mr. Bilbray. Okay, Mr. Chairman, I know my time has expired 
but in all fairness, there are a whole lot of issues that the Federal 
Government gets involved with that states, counties, cities, inter- 
national bodies could work on or whatever. Here is one where this 
really is in our lap. I just can’t perceive a state or a county or a 
coalition of states being able to address this issue, and here is one 
that I would sure like to see the Federal Government make some 
priority decisions and say this is one that only the Federal Govern- 
ment can do so the Federal Government has to be more aggressive 
on this. And I will remind all of us as we talk about other issues 
that could be handled by other agencies that we keep forgetting 
about the ones that only we can handle. And maybe we need to 
focus on these items that other people can’t handle. So I appreciate 
your testimony and yield back, Mr. Chairman. 

Chairman Miller. Thank you. Mr. Rothman was here and I 
think will be back shortly. I will recognize him out of turn if he 
returns for a first round rather than for a second round. 

In What Circumstances Should ASPs Be Deployed? 

The Chair of the Appropriations Subcommittee for Homeland Se- 
curity is my colleague from North Carolina, David Price. His dis- 
trict adjoins mine. In fact, they kind of wrap around each other like 
a couple playing Twister. If he should ask me in what cir- 
cumstances the new technology of the ASP should be deployed, 
what should I tell him? What are the markers for when it is reli- 
able enough to replace the PVTs? Should it never replace the 
PVTs? Should it always be used unless there is some breakthrough 
that we haven’t seen yet, some improvement in the technology? 
Should it always be used for secondary screening after a PVT has 
identified something to be concerned about? Or is this worth it at 
all? If the PVTs seem to be working, there is some disruption of 
having a truck pull over to the side and going over it by hand or 
cargo. But is this worth it? Mr. Aloise. 

Mr. Aloise. Well, our position has consistently been that nothing 
should be acquired in large scale until all the testing is complete, 
all the results are in, we know whether the system works or not, 
a cost benefit analysis has been done, and we can see whether it 
is worth it given all the other demands we have in our architec- 
ture, all the gaps in that architecture. After that cost benefit anal- 
ysis is done, we will have a better answer to know whether it is 
worth to pursue with the ASPs versus the PVTs. Then the question 
is where? Does it make sense more in secondary? Does it make 
sense in other places? 
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So our position, again, we need to get the testing done, all of it 
done, we need to do the cost benefit analysis. And until then we 
really don’t have enough information to make this decision on. 

Chairman Miller. Well, the original idea was to replace PVTs 
with the ASPs. Given what we know, even assuming that the test- 
ing improves, we figure out what all the problems are so that it 
doesn’t sort of go off without — it doesn’t become non-functional 
without us noticing it until after a lot of trucks have gone through 
or it doesn’t produce an amount of false positives that is going to 
shut down major ports for a significant period of time. After we 
have worked out those kinks, does it still make sense at all for it 
to be a primary detection device instead of the PVTs? 

Mr. Aloise. We have our doubts. 

Chairman Miller. Mr. Owen. 

Mr. Owen. Well, again, from our perspective, assuming it meets 
all of the technical and scientific endeavors and the machines work, 
then we have a serious concern about the operation and mainte- 
nance costs, the tail that will come with each one of these devices. 
As you mentioned in your opening statement, we are looking at 
perhaps as much as five times or more what it costs to currently 
operate it. So we also are waiting anxiously to see the cost benefit 
analysis as well as the life cycle costs to see if we can even as an 
agency afford these devices should they continue to develop and 
mature and be proven scientifically to work. 

Chairman Miller. Dr. Hagan. 

Dr. Hagan. I agree with the GAO on this. Until we get the cost 
benefit analysis and a lot of other data, by the way, that goes into 
what we call an Acquisition Review Board, it is really not prudent 
to try to presuppose the result of that. We absolutely adhere to the 
notion that doing the testing and doing the validation, doing the 
cost benefit analysis, life cycle cost estimates, all that has to be 
done and fed into this process. We have a very rigorous process 
within the department to do that. I wouldn’t want to speculate 
about whether it is going to make sense in primary or secondary, 
either or both. But we at DNDO at least are indifferent to that. 
The point is it has to be driven by, you know, real data and anal- 
ysis. 

Chairman Miller. Three years ago, Vayl Oxford, who was then 
Director of DNDO, said the priority for the first year is to get units 
out immediately. That seems like not the best idea right now. Does 
anyone disagree with that? Does anyone agree still with what Vayl 
Oxford said then, three years ago? Not a priority to get this in 
place immediately? Okay. 

When do we expect we will get the cost benefit analysis? 
Expectations of a Cost-benefit Analysis on ASPs 

Dr. Hagan. The cost benefit analysis is — we are actually strug- 
gling with this right now because of the helium-3 shortage and the 
impact of that. The cost benefit analysis of course depends on what 
the cost of the system is. We will have to most likely replace to he- 
lium-3 neutron detectors with some alternative. At this point, we 
don’t know what that alternative is. As I said before, we have sev- 
eral potential candidates for that. But until we know what that is, 
it is going to be difficult to generate a cost benefit analysis because 
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we won’t know what the cost of the systems is. It may be possible 
to work around that, but frankly we don’t know that yet. We are 
still looking at all the various options. We can try to get to a CBA, 
cost benefit analysis, but without that information and of course 
without the results from the operational test, we can’t do that. 

Chairman Miller. Dr. Persons, actually, most of my questions I 
have directed initially to Mr. Aloise and he used the first person 
plural, and then you nodded. So I didn’t direct an additional ques- 
tion at you, but you have worked with many new technologies. Are 
there best practices? Are there lessons to be learned for how to de- 
velop new practices, how to develop and deploy new technologies 
and have those lessons, have those best practices been applied with 
respect to this technology? 

Dr. Persons. Yes, sir. I worked with — coming out of an environ- 
ment where, if you are familiar with the Defense Advance Research 
Projects Agency, it is the idea about rapid technology development, 
in the absence of requirements that is normally for a very high-risk 
type, high-reward research and development paradigm. And this 
particular case, as you mentioned earlier, there is sort of some rel- 
atively mature technologies, it was just relatively immature, the 
concept of operations it was going to be asked to be placed under, 
in other words, a port operation with high volume and so on. 

And so I think some of the lessons learned from this is what you 
put your finger on just a moment ago, was what I would call the 
mistake of scheduling invention, in other words, setting artificial 
deadlines. We want to get these out and then we will worry about 
sort of the technology development later on. GAO best practices for 
many years on this topic have shown that that is a mistake to do 
that. You want to risk mitigate the way you develop your programs 
by investing and particularly empowering your science and tech- 
nology folks to do that and do that outside a procurement oper- 
ation. Even though there needs to be close ties there, it still needs 
to be this idea that the science and tech folks can do the technology 
and development, and the procurement folks can worry about how 
to do the procurement 

And so keeping that sort of balance of power, doing that, the re- 
quirements definitions is hugely important. I think in this case, the 
customer ultimately was CBP and what were they going to be com- 
fortable with in an operational paradigm? Was the con-op really 
going to be an ASP and primary alone in solo mode, or is there 
going to be a reduction in their manpower requirements and so on. 
or was it an SNM detection paradigm alone? Those are some issues 
that drive requirements through CBP out to the end of the pro- 
gram. 

And then thirdly, of course, the independent testing and evalua- 
tion, we have had a lot of discussion on that. Again, the key thing 
is there is the independence of that and how that is done. 

Chairman Miller. Okay. 

Dr. Persons. And so we have been pleased to see DNDO talking 
about interacting with DHS’s OT&E office and so on 

Chairman Miller. Okay. 

Dr. Persons. — at this phase. 

Chairman Miller. One of the lessons learned is that, at some 
point, you look for an off-ramp that if a technology seemed like a 
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good idea but just didn’t work, that you determine is not a matter 
of just working out kinks, but this was not as good idea as it 
seemed. Are there measures for when you should look for an off- 
ramp? 

Dr. Persons. Right. Well, GAO best practices recommend the 
use of readiness levels, technology readiness levels and so on. I 
think one of the things here that I think could perhaps be used is 
the idea of software readiness levels and so on because software an 
algorithms are such a critical part. I think that has manifested 
itself in the recent conclusion of the field validation tests and so 
on. 

So you have these metrics for doing that, and I think organiza- 
tionally as I believe DHS likely has is some idea about what is an 
acceptable risk of this technology or this system for deployment. 
And there are ways to measure that and just give your managers 
informed decisions so that they can determine the risk of some- 
thing, whether or not it needs to move forward and — mode. 

Chairman Miller. Well that is not quite a Dana Rohrabacher 
five minutes, but I am over the traditional five minutes. And with 
that I recognize Dr. Broun for five minutes or more. 

Metrics and Timelines for Making Decisions About 

ASPs 

Mr. Broun. Thank you, Chairman, and I am going to dovetail 
back into what the Chairman was asking. I would like to know the 
metrics. It seems to me that we keep fooling around here and have 
excuses of not implementing a technology, and not only a valid 
technology that is not only useful but may be preferable over our 
current technology, but there are other things that CBP is already 
doing. And going back to the Chairman, you are talking about an 
off-ramp, it seems to me we are getting to the point of fishing or 
cutting bait or one of the two. I mean, we have got to do something. 
If you all would give us some metrics about when we can expect 
the decision-making process, if you can help us to understand at 
what point do we need to pull the plug on the ASP program or im- 
plement something else or what have you with all of you, and I 
would like to — because I don’t have a good feeling about this, 
frankly. I don’t know about other Members but I would appreciate 
it if you all would give us some metrics or if you have them, please 
tell me right now. Mr. Aloise? 

Mr. Aloise. Well, you know, DHS came up with their “significant 
increase in operational effectiveness” as their metric for going for- 
ward with this new equipment, and we from the beginning thought 
it was a pretty low bar. I mean, I will go back to what I said. You 
know, even if this ASP works, it is a marginal increase. Now, with 
the adjustments and thresholds, that margin has gotten more nar- 
row. So what GAO’s role is here to evaluate what DHS is doing and 
provide this information to the Congress, of course, and we have 
made recommendations all along to make improvements in the 
testing, in the reporting, but the bottom line, I have to go back to 
what I have said, once the testing is done, we need to do that cost 
benefit analysis and decide whether or not it is worth to go ahead 
with this. 
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Now, it has taken longer. You know, you mentioned in 2007 they 
were ready to deploy, but the Congress who has oversight and get- 
ting GAO involved has got us to where we are today. Now, where 
do we go from here? I assume a third round of field validation test- 
ing. The first round, they made adjustments. The second round, we 
may be seeing the results of those adjustments, and we don’t know 
what is going to happen in the third round. The question is, is this 
equipment worth it? And that is a decision that DHS has to make. 
And I mean, we have got a lot of information right now, and some- 
body can make a decision or they can put this back into what we 
belong — where it should have been for a long while, and that is 
R&D, in development, and see where — what is the most you can 
get out of this promising piece of equipment. But when you know, 
we started down the path or just started down the path of acquisi- 
tion, deployment and spiral development, they have called it, and 
it has got us to where we are today, where the equipment has a 
lot of problems, it doesn’t work very well right now. And to take 
a risk and replace the equipment we have got now on the borders 
that we know works with equipment we don’t know works, you are 
always going to have GAO saying, whoa, step back. We need to 
look at this again. 

So I don’t know. It is kind of a long-winded answer to your ques- 
tion, but we need more information before we could come to a con- 
clusion, and that has to be the cost benefit analysis. 

Mr. Broun. Another question with cost benefit analysis, how can 
you do one without first knowing whether the ASP was used as a 
primary screening tool or as a secondary screening tool? I think 
that needs to be determined, too, just doing that cost benefit anal- 
ysis because otherwise you are just shooting in the dark. 

Frankly, I see the private sector developing screening methods 
without the taxpayer being on the hook for them, and we have this, 
we have spent a bunch of money and a lot of it is sitting in a ware- 
house and not being deployed and we are testing. Mr. Chairman, 
I think it might be time to have that offer out from everything that 
I can see, but I don’t know. 

And I would just like to ask Mr. Owen, would you possibly agree 
with me that it is time to maybe look at other technologies? 

Mr. Owen. Well, again, as you mentioned, the PVTs do meet our 
needs right now. They are effective. We can handle the level of 
alarms that we have got there. Hopefully, with any advances 
through the energy windowing, they give us a little bit greater 
technology detection capabilities than we have right now. Whether 
we can say we should pull the plug or not, I think again from the 
CBP standpoint, the jury is still out. We need that cost benefit 
analysis. We need to see how much better these in fact perform 
over the technology that we have before we could even make a deci- 
sion as to should they just be in secondary or should they be pri- 
mary and secondary. 

But I would like to just close by saying again, through all of our 
testing, we have kept the PVTs up and running. So we have always 
had that safety net, that effective equipment that we know has 
worked so that those 27 trucks did not get through when the ma- 
chine went down. 

Mr. Broun. Thank you, Mr. Chairman. I yield back. 
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Chairman Miller. Thank you. 

Dr. Hagan. Could I respond to any of this? 

Chairman Miller. Yes. Dr. Hagan. 

Dr. Hagan. Thank you. First of all, I wanted to comment that 
the cost benefit analysis addresses both primary and secondary 
screening, so we are not pre-determining that outcome. Second, the 
ASP, what we have been talking about here primarily is about ASP 
and PVT in primary, and you know, the debate will rage on. As I 
said before, I agree that the CBA is the way to determine that, you 
know, whether that is effective or not and worth the money. 

The ASP operating in secondary is another matter. The limita- 
tions of the current system — CBP officers are using the current 
equipment as well as they possibly can. There is no question about 
that. But there are limitations to those systems in both primary 
and secondary, and let me just focus on secondary. In secondary, 
there are limitations which improvements to PVT cannot address. 
Only ASP can do that. I can’t talk about the limitations or the 
changes or improvements of ASP compared to hand-helds in an 
open session. But the differences are dramatic. And so I think this 
talk of an off-ramp would be very premature and perhaps a closed 
session would be the way to deal with that. But I would urge you 
to consider the fact that ASP has considerable application and ben- 
efit in secondary. 

Chairman Miller. It is not really my turn, but I do hold the 
gavel. Mr. Aloise, Dr. Persons, do you all have security clearances? 
Do you have that top secret security, the top? Are you familiar with 
all of the analyses of the comparative benefits of PVT and ASP and 
primary and secondary versus hand-held? Are you familiar with all 
of that as part of the work you have done? 

Mr. Aloise. We are familiar with everything we have been 
shown, yes. 

Chairman Miller. Okay. Have you asked for everything? 

Mr. Aloise. Yes, we have. 

Chairman Miller. If there is something you haven’t been shown, 
you have asked for it and not been shown it? 

Mr. Aloise. Right, and we don’t know that that is the case 
but 

Chairman Miller. Okay. Right. And you do understand that 
GAO is part of Congress and when they ask, we are asking and we 
need for you to give them anything they have asked for. 

Mr. Aloise. Absolutely. 

Chairman Miller. Okay. Assuming that you have seen every- 
thing, is there anything — if you know all the tippy-top secret stuff, 
is it really different? Is analysis a lot different in the way that Dr. 
Hagan suggests? 

Mr. Aloise. I would say this. I think you pretty much know the 
story as I have testified and GAO has reported on. 

Now, in regards to secondary, just in discussions among our- 
selves, the ASP is bigger in radiation detection size matters. So it 
is a bigger instrument than the hand-held. So just for that — you 
have longer dwell times in secondary. So there might be uses. 
There might be a possibility where it make sense to put it into sec- 
ondary, but we still don’t have all that information. 

Chairman Miller. Okay. What is a dwell time? I am sorry. 
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Mr. Aloise. You know, the container can sit a little bit longer 
in a secondary inspection, so you have a better account. 

Chairman Miller. I see. You are not trying to move things 
through. 

Mr. Aloise. You are not driving through. 

Chairman Miller. All right. I now recognize Ms. Dahlkemper. 

Ms. Dahlkemper. Staying on this line of questioning, if it was 
used in the secondary manner, how many units would we need? 

Mr. Owen. Yeah, I don’t have the date available as to how many 
secondary locations we have in all of the ports of entry, but it is 
far fewer than what we have in primary obviously. But even with 
that, the volume through secondary is relatively low. I mean, we 
have about a one percent alarm rate on the land border and about 
2.5, 2.6 percent in seaports. So we would have to again look at the 
life cycle costs, the cost benefits as to how much better does it off- 
set what it is costing us to buy this equipment versus the time sav- 
ings or the fewer officers to address the secondaries. 

Ms. Dahlkemper. Would anyone else like to address that? I don’t 
know if anyone else can give you a better — more answer on that? 

Dr. Hagan. I can give you sort of a range. Like Mr. Owen, I don’t 
have the numbers in front of me, but I believe the number of sec- 
ondary sites is on the order of 400 to 500. Not clear that one would 
put an ASP at every one of those by any means. But as Mr. Owen 
says, it is far fewer than the number of primary screening loca- 
tions. So the 

Ms. Dahlkemper. I am certain there would be certain points 
that would be much more crucial. Maybe some of the borders — I 
mean, I live near the Canadian border. Some of those ports of entry 
are much more crucial than others. 

Dr. Hagan. Right. 

Ms. Dahlkemper. I did want to ask, though, I did want to kind 
of go back to this helium-3 issue because I think this is a crucial 
issue even beyond this hearing today. Who is in charge of the inter- 
agency task force on this? 

Dr. Hagan. It is the White House. I can’t remember the name 
of the organization. I can give you the name of the person. That 
is about all I could do. 

Ms. Dahlkemper. Who is that? 

Dr. Hagan. Her name is Julie Bentz, B-e-n-t-z. 

Ms. Dahlkemper. Okay. My understanding is the ASP need 
three times the helium-3 as the PVTs? 

Dr. Hagan. The way it was designed was to have even more sen- 
sitivity for neutrons than the PVT systems did. So we put helium- 
3 tubes in the ASP design. But if we wanted to maintain the same 
level of sensitivity as the PVT for neutrons, it would require the 
same amount of helium-3. There is nothing that links the neutron 
detection technology to whether it is an ASP or a PVT. 

Ms. Dahlkemper. As it is currently designed? 

Dr. Hagan. It was designed to have more sensitivity to neutrons 
than the PVT system because more sensitivity is better. 

Ms. Dahlkemper. So how much more helium-3 is being utilized 
in an ASP than a PVT? 

Dr. Hagan. About three times more. 

Ms. Dahlkemper. About three times? 
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Dr. Hagan. What I am saying though is if we want to go back 
to the same level of sensitivity as we currently have with the PVT, 
then the amount of helium-3 that would be required for an ASP 
unit would be the same as for a PVT unit. 

Ms. Dahlkemper. Well, then why would we have to change the 
unit if the sensitivity wasn’t increased? 

Dr. Hagan. Because this is for neutron detection only. 

Ms. Dahlkemper. Okay. 

Dr. Hagan. There is another part of the system that detects 
gamma rays, and that is the part that is different. So maybe I 
should say that the difference between an ASP and a PVT is the 
gamma ray detection part. In one case, with the PVT, it is a big 
block of plastic, and in the case of an ASP it is chunks of sodium 
iodide. But in both cases, the neutron detection part is done with 
tubes of helium-3 gas. And that part is essentially the same with 
both systems. 

Ms. Dahlkemper. So you could redesign it so it would use the 
same amount of helium-3 and not really jeopardize the security? 

Dr. Hagan. Right. You wouldn’t even have to redesign it. You 
could just take the tubes out, cut the number of tubes down by 
pulling them out. 

Ms. Dahlkemper. Okay. 

Dr. Hagan. It would be simple to do that. 

Ms. Dahlkemper. Okay. That answers my question. Thank you. 
I yield back. 

Chairman Miller. Thank you. I thought there was a Steve Roth- 
man sighting, but that was perhaps a false positive. I think we now 
had sufficient questions for now at least. We may have an addi- 
tional hearing. On this, you know, we do have — all Members have 
top-secret clearance. We have in this building and in the Cannon 
Building we have — leaded walls and Members go in and have to 
drop off their Blackberries and their cell phones before they go in, 
and they bring down a cone of silence and we hear all the top-se- 
cret stuff. And if we need to do that, we will, but if we go through 
that exercise and don’t hear anything, it is going to be pretty irri- 
tating. So let us not do that right away, but before we close the 
hearing out, Dr. Hagan, can you tell us exactly where you think we 
will be on April 1? Will we have completed successful field tests, 
will there be a cost benefit analysis, where will we be on April 1? 
Three years ago the then-head of your unit said let us get this stuff 
deployed right now. Where will we be in April? 

Dr. Hagan. Because of the helium-3 issue, I cannot give you an 
answer about the cost benefit analysis. We know how important it 
is, and we may be able to do something, but without knowing how 
much the helium-3, or the neutron detection part of the system will 
cost, it makes that difficult. But we are looking at ways to try to 
get around that. Going back to field validation — well, maybe I 
should let Mr. Owen respond about the schedule. But by April, we 
will be, I suspect, in some phase of field validation or operational 
test. 

Mr. Owen. In terms of the field validation, we have those two 
critical areas that we have been discussing. We feel those have 
been addressed by DNDO. There are a few other minor issues that 
we are working through. Our objective is to restart field validation 
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in the first quarter of calendar year 2010. So within the next 
month, we should restart field validation for a third round and go 
forward from there. 

Chairman Miller. Okay. Before we bring this hearing to a close, 
I want to thank our witnesses for testifying, and I guess it is a lit- 
tle early to say I look forward to seeing you the next time. Under 
the rules of the Committee, the record will remain open for two 
weeks for additional statements from the Members and for any an- 
swers to any follow-up questions the Committee may have for the 
witnesses. The witnesses are excused and the hearing is now ad- 
journed. 

[Whereupon, at 2:27 p.m., the Subcommittee was adjourned.] 
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Responses by William K. Hagan, Acting Deputy Director, Domestic 
Nuclear Detection Office, Department of Homeland Security; 
and Todd C. Owen, Acting Deputy Assistant Commissioner, Of- 
fice of Field Operations, U.S. Customs and Border Protection, 
Department of Homeland Security 

Questions submitted by Representative Paul C. Broun 
Questions for Dr. Hagan, and Mr. Owen: 

Ql. Do you believe computer modeling through replay tools, injection studies, or re- 
gression tools are enough for certification, or do you believe real-world field test- 
ing is required ? 

Al. Certification requires that the ASP system be both effective and suitable for the 
operation it is designed to perform and that it meets the significant increase in 
operational effectiveness (SIOE) criteria. Each of the elements listed in the question 
play a part. 

Real-world testing is vital. During the ASP program the systems 
have undergone or will undergo real-world and near-real world 
testing during controlled performance tests at the Nevada Test Site 
(NTS, real-world targets of interest), Integration testing (real-world 
interface and interoperability testing), and both Field Validation 
activities and Operational Testing (real-world stream of commerce 
and Customs Officer involvement), as well as any initial deploy- 
ments. Real-world events allow the system to be exposed to a vari- 
ety of traffic, cargo and radiological conditions that cannot be dupli- 
cated in any other way. Because real-world data is so important, 
it is vital that the data relating to traffic, cargo and radiological 
conditions be collected so that the impact of certain system changes 
can be evaluated without having to re-experience all real-world en- 
vironments. For ASP systems all of this data is collected in what 
is known as ICD-1 files. 

Replay tools provide the developer with an opportunity to re-cre- 
ate the real-world sensor data from previous data collection events 
(ICD-1 files), and to explore how changes in systems parameters 
would have impacted the results from those events. Replay tools 
are not computer modeling. These types of tools are used exten- 
sively for many types of systems, especially when data collection/ 
real-world testing is expensive, dangerous, or time consuming. In 
this case, Replay tools can determine the performance of the detec- 
tor system for a previous data collection event under various sys- 
tem parameter changes, such as threshold settings. In another ex- 
ample, Replay tools are used to determine if the system can con- 
tinue to perform if one or more of its elements are faulty. Since 
ICD-1 data files are organized by detector type, the replay tools 
can be used on modified ICD-1 files that have had this data re- 
moved. These sorts of regression analysis are extremely useful and 
can be used to provide information toward a certification decision. 
Injection studies are a way of extending the types and amounts of 
threat sources into stream of commerce data (i.e., ICD-1 files) so 
that system response to a broad spectrum of sources can be esti- 
mated. Injection studies are a type of computer modeling, and 
should therefore be checked against real, controlled experiments to 
validate that the injection tools closely match the output of real 
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systems given the same input. Injection tools are useful to point to- 
ward potential improvements in systems performance, and should 
be used during continuous improvement planning — For certifi- 
cation, the best approach to confidently understanding system per- 
formance for threat sources is to perform measurements that span 
the threat space as much as practicably possible. 

Q2. Has other ASP equipment been developed outside of this process by the private 
sector ? 

A2. DNDO is aware of other vendors, outside the ASP program, that have contin- 
ued to develop spectroscopic portals. 

Q3. How much money has DNDO spent on ASP R and D to date ? 

A3. $181M. 

Q4. What is the proper role for DNDO in this process ? 

Should DNDO be simply testing moderate changes to Commer- 
cial Off The Shelf Technology, or should it be focusing on next gen- 
eration technology? 

A4. The Congressional and Executive guidance establishing DNDO, specifically the 
SAFE Port Act of 2006, the National Security Presidential Directive 43, and Home- 
land Security Presidential Directive 14, charge DNDO with maturing “trans- 
formational technologies” as well as to “develop . . . and enhance national nuclear 
and radiological capabilities.” 

Q5. Who should be testing COTS, CBP or DNDO? 

A5. Three entities within DHS cooperate in the testing of COTS (and GOTS) sys- 
tems: DNDO, the end-user (in the case of ASP this would be CBP), and S&T. CBP 
must ensure that the testing of systems incorporates realistic operational scenario, 
DNDO must ensure that the testing properly addresses the detection and identifica- 
tion requirements of the system within the GNDA and also must ensure that any 
developmental testing has been completed prior to Operational Testing. S&T is the 
DHS Operational Testing Authority and as such must assess the suitability and ef- 
fectiveness of the final system configuration in real operational environments. 

Q6. In addressing previous problems indentified in Field Validation Testing, has the 
agency changed the underlying algorithms that were tested at the Nevada Test 
Site (NTS)? 

A6. No. 

Q7. Will simply changing isotope thresholds be enough to fix these problems, or will 
algorithms need to be changed? 

A7. As demonstrated by replay and regression testing, we are convinced that 
threshold adjustments, while reducing sensitivity, will reduce the false alarm rate 
while maintaining adherence to the ASP specification. 

Q8. If algorithms are changed, will the equipment have to go back to the NTS? 

A8. If the Department determines that changes, if they are to occur, warrant a re- 
turn, to NTS, then such testing will be conducted. 

Q9. If so, how much will this cost and how long will it take? 

Is the Department only changing threshold levels (and not 
changing algorithms) simply because it does not want to go back 
to NTS? 

A9. The cost and schedule of such tests, if needed, would be dictated by the type 
of testing required. ASP thresholds were changed because the algorithm is per- 
forming well, and because no reasons to make changes to the algorithm have been 
identified. The thresholds were changed because real-world field validation showed 
that they were not previously set in the best configuration. The Department has not 
intentionally avoided making any necessary changes to the algorithm to avoid re- 
turning to NTS. 

Q10. At our previous hearing, we discussed the possibility of deploying the Low Rate 
Production (LRP) ASPs that are currently in storage in tandem with PVTs in 
order to get more data. 
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Does the Department currently have the authority to do this? 

A10. The FY 2010 Appropriations Bill, and accompanying Conference Report, ad- 
dresses low rate initial production ASP systems. Specifically, H.Rept. 111-298 states 
that, “the conferees encourage DNDO to undertake deployment of low rate initial 
production ASP systems, as appropriate, and use data from such deployments to in- 
form future portal monitor decisions.” 

Qll. Has the Department made any progress on this front ? 

All. The Department is working through the logistics of completing operational 
testing to get to Secretarial certification before making an LRIP deployment deci- 
sion. 

Q12. How much would this cost and how long would it take? 

A12. There are eight (8) LRIP systems available for deployment. Approximately $3 
million would be needed for installing and maintaining the systems for 1 year. In- 
stallation of the 8 systems would take place over a six (6) month window. Data col- 
lection and data analysis would take approximately 6-12 months and would cost 
$100K-$200K per system. Total for all activities would be less than $5 million. 

Q13. CBP issued a RFI for ASPs several years ago only to have it rescinded by 
UNDO after the office was formed. 

In retrospect, do you believe the Request For Information (RFI) 
should have gone forward, with an eventual Request for Proposal 
(RFP) after that? 

A13. Given the urgency to address perceived limitations in fielded systems at the 
time, DHS still believes that those decisions were appropriate. It is difficult to gauge 
what may have come from an RFI/RFP process from 2003. CBP believes that the 
success of any research and development program, leading to possible deployments 
of next generation equipment, must begin with requirements defined by the end 
user/operator, and robust field evaluations/development. 

Q14. Do you believe the requirements developed by DNDO will create a system that 
can stand up to a cost-benefit analysis? 

A14. An initial cost-benefit analysis, based on performance assumptions and sys- 
tems requirements developed by DHS, warranted a decision to proceed with the pro- 
gram. Performance testing has shown potential benefits for ASP systems over cur- 
rent systems, particularly in secondary screening. However, the Department will not 
make any predictions as to the results of a final cost-benefit analysis, based on per- 
formance data, until such all testing and analysis is complete 

Q15. Should the cost-benefit analysis consider upgrades in current Radioisotope 
Identification Device (RIID) technology and Polyvinyl Toluene (PVT) “energy 
windowing” as options? 

A15. No. The CBA will assess the cost and benefits of the ASP vs. the current sys- 
tem (which currently utilizes energy windowing). Should the “current system” 
change before the CBA is complete, the CBA would be adjusted accordingly. 

Q16. How is the Department responding to the recommendations of GAO and the 
National Academy of Sciences? 

A16. DNDO has appropriately incorporated or responded to the recommendations 
of both organizations as follows. 

a. Responding to the GAO 

Recommendation regarding significant improvement to 
operational effectiveness (SIOE): The GAO Recommended 
DNDO assess whether ASPs meet SIOE criteria based on valid 
comparison with PVT’s full performance potential, including use of 
energy windowing to increase sensitivity to threats. DNDO has 
structured ASP testing to validate that ASP meets SIOE criteria. 
The SIOE criteria require that the comparison be made against the 
“current system.” DNDO will and should follow this path. We 
stress that the current PVT based system contains energy 
windowing algorithms that were updated in 2007. DNDO has also 
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funded additional research at both Pacific Northwest National Lab- 
oratory and Johns Hopkins University Applied Physics Laboratory 
to further investigate the potential to improve performance of PVT 
systems based upon algorithm enhancements, including new ap- 
proaches to energy windowing. On a side note, energy windowing 
techniques may also be applied to spectroscopic systems like the 
ASP, which may enhance performance there as well. 

Recommendation to revise ASP testing and Certification 
schedules: The GAO recommended altering schedules to allow suf- 
ficient time for review and analysis of results from final testing in- 
cluding injection studies. The ASP program schedule incorporates 
time following completion of Field Validation and Operational Test- 
ing of 

ASP, before presenting ASP to the DHS Acquisition Review 
Board for approval and then to the DHS Secretary for certification. 
This time interval prior to a decision allows all stake holders to 
thoroughly review test results and for proper consideration of the 
ASP cost-benefit analysis. 

b. Responding to the NAS 

Recommended approach for testing and evaluation: The 

NAS recommended an “iterative approach with modeling and phys- 
ical testing complimenting each other.” In this approach, “theory 
and models of threat objects, radiation transport, and detector re- 
sponse” should be used to “simulate performance and predict out- 
come,” using “physical experiments to validate the predictions and 
allow critique of the models.” We believe that the ASP program, 
through development activities carried out by vendors, to test pro- 
grams executed by the government, to replay tools and injection 
tools used for analysis, has followed the spirit of this recommenda- 
tion. The performance of the ASP systems in controlled perform- 
ance tests has mirrored predictions extremely well. The issue that 
remains for ASP is one of threshold settings against real-world 
cargo which must be determined through empirical evidence. 

Recommended Approach of the Procurement Process: The 
NAS recommended that DNDO employ an “iterative” approach for 
the procurement process. Rather than a “onetime certification deci- 
sion in the near future,” the NAS recommended a continuous proc- 
ess of improvement and adaptation of the system, beginning with 
the deployment of LRIP systems. We agree with the NAS in prin- 
ciple and are evaluating this approach. This approach is consistent 
with the ASP Acquisition Plan and will be reflected in actual de- 
ployments when they begin. 

Recommended Approach for Cost-Benefit Analysis: The 

NAS Committee recommended that DHS not proceed with further 
procurement until “ASP is shown to be a favored option in the cost- 
benefit analysis.” This recommendation is incorporated into the 
governance structure of the ASP program. Additionally the NAS 
made recommendations about the nature of the CBA itself, stating 
that it “needs to include three key elements: (1) a clear statement 
of the objectives of the screening program: (2) an assessment of 
meaningful alternatives to deploying ASPs; and (3) a comprehen- 
sive, credible and transparent analysis of in-scope benefits and 
costs.” We continue to agree with the NAS that a complete CBA is 
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necessary before any decisions are made and the elements sug- 
gested by NAS are factored into the CBA. 

Q17. Setting aside the increased costs of ASPs over the current PVT/RIDD combina- 
tion, how much more money will ASP Operations and Maintenance (0 and M) 
cost CBP? 

A17. Current PVT RPM units cost approximately $12,000 per year to operate and 
maintain. While CBP has not seen the final ASP maintenance cost estimate, this 
figure is expected to range from $65-100k each year per unit. 

Q18. Given dwindling budget estimates, how will these increased O&M costs affect 
overall security ? 

A18. If no increase in O&M funds accompanies the deployment of ASPs, CBP would 
have to secure funding from other programs or staff positions. 

Q19. Will CBP even be able to operate ASPs in the current budget environment if 
they are certified ? 

A19. CBP anticipates that the appropriate O&M funding would be included should 
the ASPs be deployed. 

Q20. If the ASP systems are certified and deployed, will the Department be able to 
find cost savings? 

A20. The deployment of ASPs will result in an overall increase in life cycle costs. 
Q21. Does the ASP system require less CBP personnel to operate? 

A21. There is no appreciable savings in required personnel over the entire deployed 
architecture. In primary scanning operations, for example, land border crossings 
would not see a staff reduction. In secondary, the ASPs may require less physical 
labor and less time to operate; however, a CBP Officer will still be required to oper- 
ate the ASPs. It should be noted that the ASP will not eliminate the need for RIIDs 
which will be used for more detailed inspections and alarm adjudication. 

Only very limited reductions likewise would be seen in seaport 
deployments. 

Q22. Does the ASP system allow for greater flow of commerce, or would other factors 
at ports and border crossings still impede commerce? 

A22. No appreciable gains to the flow of commerce would be realized over the entire 
deployed architecture. At land border crossings, impediments to the flow of com- 
merce are due to non-radiation scanning processes (i.e. — passport control and anti- 
smuggling inspections). At seaports, the flow of commerce is mostly determined by 
individual terminal procedures. 
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Answers to Post-Hearing Questions 

Responses by Gene Aloise, Director, Natural Resources and Environment, U.S. Gov- 
ernment Accountability Office (GAO); Todd C. Owen, Acting Deputy Assistant 
Commissioner, Office of Field Operations, U.S. Customs and Border Protection, 
Department of Homeland Security; and William K. Hagan, Acting Deputy Direc- 
tor, Domestic Nuclear Detection Office, Department of Homeland Security 

Questions submitted by Representative Paul C. Broun 
Questions for Mr. Aloise, Dr. Hagan, and Mr. Owen: 

Ql. Do you believe computer modeling through replay tools, injection studies, or re- 
gression tools is enough for certification, or do you believe real world field test- 
ing is required ? 

Al. The current campaign of ASP testing, which started in early 2008, involved sev- 
eral different types of testing each for its own purpose. The August 2008 testing at 
the Nevada Test Site (NTS) was designed to test sensitivity of the ASPs (and PVTs) 
against actual physical sources of special nuclear materials (e.g., uranium and plu- 
tonium). For the current campaign of testing, the NTS phase of testing was particu- 
larly important given that the ASPs were expected to have an improved capability 
of detecting special nuclear materials, and, for security reasons, NTS was the only 
place where DNDO could fully physically test the ASPs capability of detecting these 
very dangerous materials. 

To this point, the field validation tests have shown an unacceptable level of false 
alarms for the presence of certain nuclear materials. To resolve this, DNDO officials 
told us that they have made changes to the thresholds of the ASP, but not to the 
underlying algorithms (or software) that allows these machines to detect radiation. 
As we testified, this change makes the ASP less capable of detecting certain nuclear 
materials, raising questions about whether the ASPs represent a significant im- 
provement in operational effectiveness over the PVTs in detecting certain nuclear 
materials. 

While DNDO officials told us that they were not making changes to the ASP algo- 
rithms, the report of the Field Validation Advisory Panel stated that “correcting al- 
gorithm issues will change ASP referral rate” suggesting that there will be future 
changes to the ASP algorithms. So, there is some confusion about exactly what 
changes will need to be made in order to successfully complete field validation test- 
ing. This represents a subtle, but important distinction — we regard “algorithm cor- 
rections” as algorithm changes and not simple threshold adjustment. 

In our view, prior to any certification decision to move forward with procurement 
of ASPs, there should be additional testing at NTS against actual physical sources 
of special nuclear materials in order to (1) understand exactly how the modified 
ASPs performance compares against the performance of PVTs in detecting certain 
nuclear materials, and (2) assure that there are not any unintended consequences 
in detection capabilities resulting from the modifications made to the ASPs since the 
August 2008 testing at NTS. 1 This is to say, at this stage in the development of 
the ASP, major systems acquisition best practices require performance testing with 
actual special nuclear material because of the expected corrections to the software 
(beyond simple threshold adjustment). 

Q2a. Has other ASP equipment been developed outside of this process by the private 
sector ? 

A2a. Based on the number of proposals submitted to DNDO’s initial RFP for the 
ASP, it is quite possible that spectroscopic radiation detection systems have been 
developed outside of DNDO sponsorship. The issue here is determining the extent 
to which those COTS vendors have focused development on spectroscopic radiation 
detection systems that could operate in a high-impact port environment (large for- 
mat and high traffic throughput) in support of CBP’s mission. Moreover, only 
DNDO’s two developers have been afforded the ability to test their systems with ac- 
tual special nuclear material at the NTS. 

Q2b. How much money has DNDO spent on ASP R&D to date ? 

A2b. DNDO would be the best source for this information. 


1 Without tests at NTS we will not have True Positive or False Negative data with the new 
software. 
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Q2c. What is the proper role for DNDO in this process? Should DNDO be simply 
testing moderate changes to Commercial off the Shelf Technology or should it 
be focusing on next generation technology? 

A2c. We believe the Secretary of DHS should determine whether DNDO should be 
a technology development or a major systems acquisition organization. If she choos- 
es to keep both functions within the organization, then the Secretary should estab- 
lish a strong separation and independent operational construct between the tech- 
nology development and major systems acquisition missions. Ultimately, DNDO’s 
goal should be toward addressing the mission needs of the end-user (CBP in the 
case of the ASP program). Organizational best practices hold that DNDO should 
regularly assess the extent to which these needs may be met with existing commer- 
cial-off-the-shelf technologies, as well as the nature and extent of development nec- 
essary for emerging or next-generation technologies. In the case of ASPs, the mis- 
sion need has become less clear based upon the lessening of the criteria for a “sig- 
nificant increase in operational effectiveness.” This is to say, the primary mission 
need DNDO put forward was to better detect certain special nuclear materials, and 
the expectation was established that ASPs would vastly outperform PVTs in this 
area as manifested by a dramatic reduction of referrals to secondary screening due 
to NORM. However, recent testing has raised questions about both the ASPs capa- 
bilities to reliably and efficiently detect these materials as well as the exact signifi- 
cance of the increase in operational effectiveness of these next generation systems 
that are 2.5 times more expensive per unit than the current generation of tech- 
nology. Moreover, it is not clear that the PVT technology has reached its limit of 
performance based upon energy windowing algorithms or other advances in mate- 
rials science for nuclear detection. 

Q2d. Who should be testing COTS, DNDO, or CBP? 

A2d. GAO sees three parts of a testing regime for these systems: (1) software devel- 
opment testing, (2) physical performance testing at NTS (supported by follow-up 
modeling and simulation work), and (3) field validation testing. In our view, the first 
is managed by DNDO and its contractors with independent testing and evaluation 
done by a federally funded research and development center (FFRDC), university 
affiliated research center (UARC), or the National Institute of Standards and Tech- 
nology (NIST). The second is also managed by DNDO in partnership with the Na- 
tional Nuclear Security Administration’s staff at NTS as well as the OT&E group 
of DHS and/or NIST (simulation work can be supported by national labs or UARCs). 
The third should be managed by CBP with support from DHS OT&E as the end- 
user since they are responsible for screening operations at a working port of entry. 

Questions for Mr. Aloise and Dr. Hagan: 

Q3a. In addressing previous problems identified in Field Validation Testing, has the 
agency changed underlying algorithms that were tested at the Nevada Test Site 
(NTS)? 

A3a. As discussed above, while DNDO officials told us that they were not making 
changes to the ASP algorithms, the report of the Field Validation Advisory Panel 
stated that “correcting algorithm issues will change ASP referral rate” suggesting 
that there will be future changes to the ASP algorithms. So, there is some confusion 
about exactly what changes will need to be made in order to successfully complete 
field validation testing. 

Q3b. Will simply changing isotope thresholds be enough to fix these problems, or will 
algorithms need to be changed? 

A3b. Only additional testing can answer this. However, once the problem with false 
alarms is fixed, the ability of the ASPs to detect certain nuclear materials will be 
diminished. Understanding whether the ASP can detected certain nuclear materials 
better than the PVTs will be an important part of deciding whether to move forward 
with procurement of the ASPs. 

Q3c. If algorithms are changed, will the equipment have to go back to the NTS? 

A3c. In our view, prior to any decision to move forward with procurement of ASPs, 
there should be additional testing at NTS against actual physical sources of special 
nuclear materials in order to (1) understand exactly how the modified ASPs per- 
formance compares against the performance of PVTs in detecting certain nuclear 
materials, and (2) assure that there are not any unintended consequences in detec- 
tion capabilities resulting from the modifications made to the ASPs since the August 
2008 testing at NTS. 
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Q3d. If so, how much will this cost and how long will it take ? 

A3d. DNDO would be the best source for this information. However, we were pre- 
viously told by DNDO that NTS testing costs about $500,000 per week. In our view, 
this testing, while expensive, would be worthwhile prior to DNDO committing bil- 
lions of dollars toward procurement of the ASPs. 

Q3e. Is the Department only changing threshold levels (and not changing the algo- 
rithms) simply because it does not want to go back to NTS? 

A3e. DHS would be the best source for this information. 
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